[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:06:28 dist-test-slave-j889 krb5kdc[417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:06:28 dist-test-slave-j889 krb5kdc[417](info): set up 2 sockets
Feb 03 21:06:28 dist-test-slave-j889 krb5kdc[417](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260203 21:06:29.977144 410 mini_kdc.cc:121] Time spent starting KDC: real 1.815s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:06:30 dist-test-slave-j889 krb5kdc[417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:06:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:06:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:30.106033 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:42273
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:42507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:42273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:30.207953 433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:30.208196 433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:30.208269 433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:30.211476 433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:06:30.211537 433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:30.211570 433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:30.211593 433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:06:30.211624 433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:06:30.215701 433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:42507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:42273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:42273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:30.216708 433 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:30.217556 433 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:30.223582 439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:30.223678 441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:30.223691 438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:30.223894 433 server_base.cc:1060] running on GCE node
I20260203 21:06:30.224351 433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:30.225286 433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:30.226428 433 hybrid_clock.cc:648] HybridClock initialized: now 1770152790226394 us; error 48 us; skew 500 ppm
Feb 03 21:06:30 dist-test-slave-j889 krb5kdc[417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:30.228843 433 init.cc:377] Logged in from keytab as kudu/127.0.102.190@KRBTEST.COM (short username kudu)
I20260203 21:06:30.229887 433 webserver.cc:492] Webserver started at http://127.0.102.190:39617/ using document root <none> and password file <none>
I20260203 21:06:30.230434 433 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:30.230481 433 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:30.230665 433 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:30.232369 433 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "ddf399ee1efd4bf5b51671978baa0894"
format_stamp: "Formatted at 2026-02-03 21:06:30 on dist-test-slave-j889"
server_key: "63432f75e03652e177d4bfbab5fe9575"
server_key_iv: "4ff8b51fb5a955d1b64865963d14869a"
server_key_version: "encryptionkey@0"
I20260203 21:06:30.232856 433 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "ddf399ee1efd4bf5b51671978baa0894"
format_stamp: "Formatted at 2026-02-03 21:06:30 on dist-test-slave-j889"
server_key: "63432f75e03652e177d4bfbab5fe9575"
server_key_iv: "4ff8b51fb5a955d1b64865963d14869a"
server_key_version: "encryptionkey@0"
I20260203 21:06:30.236186 433 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260203 21:06:30.238373 448 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:30.239501 433 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:06:30.239624 433 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "ddf399ee1efd4bf5b51671978baa0894"
format_stamp: "Formatted at 2026-02-03 21:06:30 on dist-test-slave-j889"
server_key: "63432f75e03652e177d4bfbab5fe9575"
server_key_iv: "4ff8b51fb5a955d1b64865963d14869a"
server_key_version: "encryptionkey@0"
I20260203 21:06:30.239717 433 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:30.257474 433 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:30.260787 433 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:30.261014 433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:30.268163 500 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:42273 every 8 connection(s)
I20260203 21:06:30.268177 433 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:42273
I20260203 21:06:30.269336 433 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:06:30.271255 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 433
I20260203 21:06:30.271407 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:06:30.272717 410 external_mini_cluster.cc:1468] Setting key 4969055fca1c78cb5dfe95909fd4bf5f
I20260203 21:06:30.272882 501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:30.279697 501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894: Bootstrap starting.
I20260203 21:06:30.282501 501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:30.283334 501 log.cc:826] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:30.285579 501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894: No bootstrap required, opened a new log
Feb 03 21:06:30 dist-test-slave-j889 krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:30.288463 501 raft_consensus.cc:359] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf399ee1efd4bf5b51671978baa0894" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42273 } }
I20260203 21:06:30.288681 501 raft_consensus.cc:385] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:30.288748 501 raft_consensus.cc:740] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf399ee1efd4bf5b51671978baa0894, State: Initialized, Role: FOLLOWER
I20260203 21:06:30.289187 501 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf399ee1efd4bf5b51671978baa0894" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42273 } }
I20260203 21:06:30.289340 501 raft_consensus.cc:399] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:06:30.289405 501 raft_consensus.cc:493] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:06:30.289499 501 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:30.290393 501 raft_consensus.cc:515] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf399ee1efd4bf5b51671978baa0894" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42273 } }
I20260203 21:06:30.290764 501 leader_election.cc:304] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddf399ee1efd4bf5b51671978baa0894; no voters:
I20260203 21:06:30.291118 501 leader_election.cc:290] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:06:30.291556 506 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:30.291805 506 raft_consensus.cc:697] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [term 1 LEADER]: Becoming Leader. State: Replica: ddf399ee1efd4bf5b51671978baa0894, State: Running, Role: LEADER
I20260203 21:06:30.292132 506 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf399ee1efd4bf5b51671978baa0894" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42273 } }
I20260203 21:06:30.292197 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:30.275463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.0.102.190:42273)
0203 21:06:30.276002 (+ 539us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:30.276015 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:30.276377 (+ 362us) server_negotiation.cc:408] Connection header received
0203 21:06:30.279147 (+ 2770us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:30.279164 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:30.279460 (+ 296us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:30.279722 (+ 262us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:30.281950 (+ 2228us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:30.282703 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:30.283557 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:30.283792 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:30.287205 (+ 3413us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:30.287225 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:30.287235 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:30.287271 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:30.289105 (+ 1834us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:30.289646 (+ 541us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:30.289651 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:30.289655 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:30.289732 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:30.290064 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:30.290067 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:30.290069 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:30.290397 (+ 328us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:30.290613 (+ 216us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:30.291110 (+ 497us) server_negotiation.cc:300] Negotiation successful
0203 21:06:30.291364 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":117,"threads_started":1}
I20260203 21:06:30.292268 501 sys_catalog.cc:565] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:06:30.294330 507 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddf399ee1efd4bf5b51671978baa0894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf399ee1efd4bf5b51671978baa0894" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42273 } } }
I20260203 21:06:30.294673 507 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [sys.catalog]: This master's current role is: LEADER
I20260203 21:06:30.294598 508 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf399ee1efd4bf5b51671978baa0894. Latest consensus state: current_term: 1 leader_uuid: "ddf399ee1efd4bf5b51671978baa0894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf399ee1efd4bf5b51671978baa0894" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42273 } } }
I20260203 21:06:30.294903 508 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894 [sys.catalog]: This master's current role is: LEADER
W20260203 21:06:30.297011 520 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260203 21:06:30.297072 520 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260203 21:06:30.297153 521 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:06:30.297995 521 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:06:30.303061 521 catalog_manager.cc:1357] Generated new cluster ID: 9c40d8f21f84415fb1696091d0b0b447
I20260203 21:06:30.303130 521 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:06:30.317833 521 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:06:30.319036 521 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:06:30.326050 521 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ddf399ee1efd4bf5b51671978baa0894: Generated new TSK 0
I20260203 21:06:30.326864 521 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:30.372124 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:42273
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:42507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:30.479722 529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:30.479933 529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:30.479996 529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:30.483202 529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:30.483264 529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:30.483363 529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:06:30.487504 529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:42507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:42273
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:30.488559 529 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:30.489420 529 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:30.495755 535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:30.495765 534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:30.495764 537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:30.496061 529 server_base.cc:1060] running on GCE node
I20260203 21:06:30.496606 529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:30.497234 529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:30.498395 529 hybrid_clock.cc:648] HybridClock initialized: now 1770152790498381 us; error 29 us; skew 500 ppm
Feb 03 21:06:30 dist-test-slave-j889 krb5kdc[417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:30.500998 529 init.cc:377] Logged in from keytab as kudu/127.0.102.129@KRBTEST.COM (short username kudu)
I20260203 21:06:30.502043 529 webserver.cc:492] Webserver started at http://127.0.102.129:43875/ using document root <none> and password file <none>
I20260203 21:06:30.502573 529 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:30.502621 529 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:30.502804 529 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:30.504499 529 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "ffc8b2255e034725acc8f2c88b357566"
format_stamp: "Formatted at 2026-02-03 21:06:30 on dist-test-slave-j889"
server_key: "2cd945899656c00c077a226ee8ebf666"
server_key_iv: "2a94957b028e2041cf78b7e2c0593578"
server_key_version: "encryptionkey@0"
I20260203 21:06:30.505007 529 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "ffc8b2255e034725acc8f2c88b357566"
format_stamp: "Formatted at 2026-02-03 21:06:30 on dist-test-slave-j889"
server_key: "2cd945899656c00c077a226ee8ebf666"
server_key_iv: "2a94957b028e2041cf78b7e2c0593578"
server_key_version: "encryptionkey@0"
I20260203 21:06:30.508249 529 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260203 21:06:30.510427 544 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:30.511418 529 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:06:30.511518 529 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "ffc8b2255e034725acc8f2c88b357566"
format_stamp: "Formatted at 2026-02-03 21:06:30 on dist-test-slave-j889"
server_key: "2cd945899656c00c077a226ee8ebf666"
server_key_iv: "2a94957b028e2041cf78b7e2c0593578"
server_key_version: "encryptionkey@0"
I20260203 21:06:30.511601 529 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:30.531100 529 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:30.533704 529 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:30.533879 529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:30.534435 529 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:30.535295 529 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:30.535342 529 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:30.535394 529 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:30.535422 529 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:30.544158 529 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:33827
I20260203 21:06:30.544251 657 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:33827 every 8 connection(s)
I20260203 21:06:30.545173 529 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:06:30.548002 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 529
I20260203 21:06:30.548110 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:06:30.548362 410 external_mini_cluster.cc:1468] Setting key 06f36fa3bc7cea262d500844c2c1dc4c
Feb 03 21:06:30 dist-test-slave-j889 krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.102.130@KRBTEST.COM; defaulting to no policy
I20260203 21:06:30.558535 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:30.546917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:45423 (local address 127.0.102.190:42273)
0203 21:06:30.547057 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:30.547061 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:30.547739 (+ 678us) server_negotiation.cc:408] Connection header received
0203 21:06:30.548799 (+ 1060us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:30.548803 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:30.548875 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:30.548982 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:30.551037 (+ 2055us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:30.551555 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:30.552400 (+ 845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:30.552532 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:30.555110 (+ 2578us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:30.555134 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:30.555136 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:30.555165 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:30.556492 (+ 1327us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:30.557140 (+ 648us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:30.557143 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:30.557145 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:30.557194 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:30.557479 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:30.557481 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:30.557482 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:30.557637 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:30.557708 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:30.558285 (+ 577us) server_negotiation.cc:300] Negotiation successful
0203 21:06:30.558401 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260203 21:06:30.559252 660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:30.547168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42273 (local address 127.0.102.129:45423)
0203 21:06:30.547597 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:30.547634 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:30.548557 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:30.549151 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:30.549163 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:30.549591 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:30.550855 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:30.550874 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:30.551680 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:30.551684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:30.552295 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:30.552305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:30.552527 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:30.553118 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:30.553159 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:30.554916 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:30.556662 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:30.556669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:30.556684 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:30.557036 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:30.557286 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:30.557289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:30.557291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:30.557402 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:30.557716 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:30.557721 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:30.557975 (+ 254us) client_negotiation.cc:769] Sending connection context
0203 21:06:30.558215 (+ 240us) client_negotiation.cc:240] Negotiation successful
0203 21:06:30.558457 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.130@KRBTEST.COM" created.
I20260203 21:06:30.560452 658 heartbeater.cc:344] Connected to a master server at 127.0.102.190:42273
I20260203 21:06:30.560729 658 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:30.561395 658 heartbeater.cc:507] Master 127.0.102.190:42273 requested a full tablet report, sending...
I20260203 21:06:30.563130 464 ts_manager.cc:194] Registered new tserver with Master: ffc8b2255e034725acc8f2c88b357566 (127.0.102.129:33827)
I20260203 21:06:30.564849 464 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:45423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:30.595597 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:42273
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:42507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:30.703644 665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:30.703872 665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:30.703962 665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:30.707330 665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:30.707408 665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:30.707516 665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:06:30.711825 665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:42507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:42273
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:30.712982 665 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:30.713869 665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:30.720168 673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:30.720183 671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:30.720326 665 server_base.cc:1060] running on GCE node
W20260203 21:06:30.720209 670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:30.720919 665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:30.721489 665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:30.722657 665 hybrid_clock.cc:648] HybridClock initialized: now 1770152790722632 us; error 32 us; skew 500 ppm
Feb 03 21:06:30 dist-test-slave-j889 krb5kdc[417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:30.725286 665 init.cc:377] Logged in from keytab as kudu/127.0.102.130@KRBTEST.COM (short username kudu)
I20260203 21:06:30.726248 665 webserver.cc:492] Webserver started at http://127.0.102.130:44983/ using document root <none> and password file <none>
I20260203 21:06:30.726758 665 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:30.726807 665 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:30.726985 665 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:30.728662 665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "13465959f40f4786b15ab6edb437ee92"
format_stamp: "Formatted at 2026-02-03 21:06:30 on dist-test-slave-j889"
server_key: "791288052a1770116d4dd2a3200fa41a"
server_key_iv: "21f919becf8ee4a6d3d9344d1d8cd92c"
server_key_version: "encryptionkey@0"
I20260203 21:06:30.729113 665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "13465959f40f4786b15ab6edb437ee92"
format_stamp: "Formatted at 2026-02-03 21:06:30 on dist-test-slave-j889"
server_key: "791288052a1770116d4dd2a3200fa41a"
server_key_iv: "21f919becf8ee4a6d3d9344d1d8cd92c"
server_key_version: "encryptionkey@0"
I20260203 21:06:30.732427 665 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260203 21:06:30.734584 680 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:30.735527 665 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:06:30.735631 665 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "13465959f40f4786b15ab6edb437ee92"
format_stamp: "Formatted at 2026-02-03 21:06:30 on dist-test-slave-j889"
server_key: "791288052a1770116d4dd2a3200fa41a"
server_key_iv: "21f919becf8ee4a6d3d9344d1d8cd92c"
server_key_version: "encryptionkey@0"
I20260203 21:06:30.735720 665 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:30.761808 665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:30.764747 665 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:30.764937 665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:30.765503 665 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:30.766376 665 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:30.766424 665 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:30.766474 665 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:30.766503 665 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:30.776012 665 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:34427
I20260203 21:06:30.776043 793 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:34427 every 8 connection(s)
I20260203 21:06:30.777001 665 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:06:30.781209 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 665
I20260203 21:06:30.781318 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:06:30.781522 410 external_mini_cluster.cc:1468] Setting key 5338a22f003d5a3b4767f8890a258e30
Feb 03 21:06:30 dist-test-slave-j889 krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.102.131@KRBTEST.COM; defaulting to no policy
I20260203 21:06:30.790973 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:30.778789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:39445 (local address 127.0.102.190:42273)
0203 21:06:30.778922 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:30.778926 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:30.779761 (+ 835us) server_negotiation.cc:408] Connection header received
0203 21:06:30.781235 (+ 1474us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:30.781239 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:30.781305 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:30.781396 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:30.783425 (+ 2029us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:30.783930 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:30.784667 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:30.784857 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:30.787340 (+ 2483us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:30.787364 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:30.787369 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:30.787402 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:30.788971 (+ 1569us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:30.789499 (+ 528us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:30.789503 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:30.789505 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:30.789564 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:30.789847 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:30.789851 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:30.789853 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:30.790065 (+ 212us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:30.790177 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:30.790708 (+ 531us) server_negotiation.cc:300] Negotiation successful
0203 21:06:30.790837 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260203 21:06:30.791638 796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:30.779060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42273 (local address 127.0.102.130:39445)
0203 21:06:30.779580 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:30.779620 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:30.781009 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:30.781627 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:30.781639 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:30.782236 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:30.783278 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:30.783293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:30.784057 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:30.784060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:30.784546 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:30.784555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:30.784798 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:30.785443 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:30.785472 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:30.787158 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:30.789120 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:30.789124 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:30.789138 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:30.789395 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:30.789656 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:30.789658 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:30.789660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:30.789759 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:30.790173 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:30.790179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:30.790447 (+ 268us) client_negotiation.cc:769] Sending connection context
0203 21:06:30.790652 (+ 205us) client_negotiation.cc:240] Negotiation successful
0203 21:06:30.790906 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.131@KRBTEST.COM" created.
I20260203 21:06:30.792910 794 heartbeater.cc:344] Connected to a master server at 127.0.102.190:42273
I20260203 21:06:30.793179 794 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:30.793709 794 heartbeater.cc:507] Master 127.0.102.190:42273 requested a full tablet report, sending...
I20260203 21:06:30.794898 464 ts_manager.cc:194] Registered new tserver with Master: 13465959f40f4786b15ab6edb437ee92 (127.0.102.130:34427)
I20260203 21:06:30.795666 464 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.130@KRBTEST.COM'} at 127.0.102.130:39445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:30.829043 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:42273
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:42507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:30.935946 801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:30.936170 801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:30.936226 801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:30.939456 801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:30.939523 801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:30.939604 801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:06:30.943670 801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:42507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:42273
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:30.944684 801 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:30.945452 801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:30.951839 809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:30.951840 807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:30.952070 806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:30.952244 801 server_base.cc:1060] running on GCE node
I20260203 21:06:30.952657 801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:30.953227 801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:30.954388 801 hybrid_clock.cc:648] HybridClock initialized: now 1770152790954370 us; error 36 us; skew 500 ppm
Feb 03 21:06:30 dist-test-slave-j889 krb5kdc[417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:30.956912 801 init.cc:377] Logged in from keytab as kudu/127.0.102.131@KRBTEST.COM (short username kudu)
I20260203 21:06:30.957937 801 webserver.cc:492] Webserver started at http://127.0.102.131:40239/ using document root <none> and password file <none>
I20260203 21:06:30.958473 801 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:30.958523 801 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:30.958694 801 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:30.960377 801 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "de9316063ecf40a097a6e74c11ada5fd"
format_stamp: "Formatted at 2026-02-03 21:06:30 on dist-test-slave-j889"
server_key: "f5b8f6d20f538362dd250b1d14800658"
server_key_iv: "9d5e9e221ad72d7c76f00e7f9095fcbb"
server_key_version: "encryptionkey@0"
I20260203 21:06:30.960852 801 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "de9316063ecf40a097a6e74c11ada5fd"
format_stamp: "Formatted at 2026-02-03 21:06:30 on dist-test-slave-j889"
server_key: "f5b8f6d20f538362dd250b1d14800658"
server_key_iv: "9d5e9e221ad72d7c76f00e7f9095fcbb"
server_key_version: "encryptionkey@0"
I20260203 21:06:30.964306 801 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260203 21:06:30.966470 816 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:30.967499 801 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:06:30.967598 801 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "de9316063ecf40a097a6e74c11ada5fd"
format_stamp: "Formatted at 2026-02-03 21:06:30 on dist-test-slave-j889"
server_key: "f5b8f6d20f538362dd250b1d14800658"
server_key_iv: "9d5e9e221ad72d7c76f00e7f9095fcbb"
server_key_version: "encryptionkey@0"
I20260203 21:06:30.967684 801 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:30.981158 801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:30.984177 801 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:30.984382 801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:30.985002 801 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:30.985936 801 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:30.985991 801 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:30.986050 801 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:30.986083 801 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:30.996837 801 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:33643
I20260203 21:06:30.996861 929 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:33643 every 8 connection(s)
I20260203 21:06:30.997897 801 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
I20260203 21:06:31.004215 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 801
I20260203 21:06:31.004304 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:06:31.004503 410 external_mini_cluster.cc:1468] Setting key df92dcf82579a948f70f21373eaa2c72
Feb 03 21:06:31 dist-test-slave-j889 krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:31.010048 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:30.999642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:39513 (local address 127.0.102.190:42273)
0203 21:06:30.999789 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:30.999794 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:31.000441 (+ 647us) server_negotiation.cc:408] Connection header received
0203 21:06:31.001376 (+ 935us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:31.001380 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:31.001443 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:31.001516 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:31.003210 (+ 1694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:31.003716 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:31.004412 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:31.004602 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:31.007163 (+ 2561us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:31.007179 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:31.007181 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:31.007208 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:31.008407 (+ 1199us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:31.008882 (+ 475us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:31.008884 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:31.008886 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:31.008927 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:31.009143 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:31.009145 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:31.009146 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:31.009292 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:31.009363 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:31.009786 (+ 423us) server_negotiation.cc:300] Negotiation successful
0203 21:06:31.009876 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260203 21:06:31.010643 932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:30.999869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42273 (local address 127.0.102.131:39513)
0203 21:06:31.000296 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:31.000332 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:31.001174 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:31.001695 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:31.001704 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:31.002125 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:31.003035 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:31.003050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:31.003843 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:31.003846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:31.004265 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:31.004274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:31.004517 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:31.005156 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:31.005200 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:31.006889 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:31.008510 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:31.008514 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:31.008523 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:31.008808 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:31.008990 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:31.008992 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:31.008994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:31.009078 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:31.009360 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:31.009365 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:31.009595 (+ 230us) client_negotiation.cc:769] Sending connection context
0203 21:06:31.009753 (+ 158us) client_negotiation.cc:240] Negotiation successful
0203 21:06:31.009975 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":101,"threads_started":1}
I20260203 21:06:31.011756 930 heartbeater.cc:344] Connected to a master server at 127.0.102.190:42273
I20260203 21:06:31.011968 930 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:31.012451 930 heartbeater.cc:507] Master 127.0.102.190:42273 requested a full tablet report, sending...
I20260203 21:06:31.013383 464 ts_manager.cc:194] Registered new tserver with Master: de9316063ecf40a097a6e74c11ada5fd (127.0.102.131:33643)
I20260203 21:06:31.013887 464 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.131@KRBTEST.COM'} at 127.0.102.131:39513
I20260203 21:06:31.018352 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 03 21:06:31 dist-test-slave-j889 krb5kdc[417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152791, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 03 21:06:31 dist-test-slave-j889 krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152791, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.102.129@KRBTEST.COM
I20260203 21:06:31.039789 937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:31.032147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.0.102.129:33827)
0203 21:06:31.032394 (+ 247us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:31.032400 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:31.032423 (+ 23us) server_negotiation.cc:408] Connection header received
0203 21:06:31.032528 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:31.032534 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:31.032737 (+ 203us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:31.032865 (+ 128us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:31.033595 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:31.034291 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:31.034898 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:31.035059 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:31.036958 (+ 1899us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:31.036982 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:31.036995 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:31.037019 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:31.038541 (+ 1522us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:31.038847 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:31.038852 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:31.038853 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:31.038899 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:31.039038 (+ 139us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:31.039040 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:31.039041 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:31.039213 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:31.039376 (+ 163us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:31.039474 (+ 98us) server_negotiation.cc:300] Negotiation successful
0203 21:06:31.039607 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":56,"threads_started":1}
W20260203 21:06:31.040690 572 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46506
Feb 03 21:06:31 dist-test-slave-j889 krb5kdc[417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 03 21:06:31 dist-test-slave-j889 krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.129@KRBTEST.COM
I20260203 21:06:31.063395 937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:31.055908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.0.102.129:33827)
0203 21:06:31.056039 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:31.056043 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:31.056124 (+ 81us) server_negotiation.cc:408] Connection header received
0203 21:06:31.056273 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:31.056276 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:31.056326 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:31.056397 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:31.057216 (+ 819us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:31.057712 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:31.058331 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:31.058533 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:31.060523 (+ 1990us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:31.060541 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:31.060543 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:31.060569 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:31.061698 (+ 1129us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:31.062042 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:31.062046 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:31.062048 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:31.062094 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:31.062288 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:31.062291 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:31.062293 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:31.062435 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:31.063097 (+ 662us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:31.063161 (+ 64us) server_negotiation.cc:300] Negotiation successful
0203 21:06:31.063274 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260203 21:06:31.064376 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 529
I20260203 21:06:31.069473 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 665
I20260203 21:06:31.076480 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 801
I20260203 21:06:31.081274 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 433
2026-02-03T21:06:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2946 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:06:31 dist-test-slave-j889 krb5kdc[944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:06:31 dist-test-slave-j889 krb5kdc[944](info): set up 2 sockets
Feb 03 21:06:31 dist-test-slave-j889 krb5kdc[944](info): commencing operation
krb5kdc: starting...
W20260203 21:06:32.920262 410 mini_kdc.cc:121] Time spent starting KDC: real 1.818s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:06:33 dist-test-slave-j889 krb5kdc[944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:06:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:06:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:33.055732 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:43993
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:43993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:33.158429 960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:33.158667 960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:33.158726 960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:33.161998 960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:06:33.162065 960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:33.162088 960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:33.162120 960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:06:33.162150 960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:06:33.166426 960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:43993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:43993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:33.167497 960 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:33.168331 960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:33.173698 965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:33.173703 968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:33.173843 960 server_base.cc:1060] running on GCE node
W20260203 21:06:33.173763 966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:33.174464 960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:33.175333 960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:33.176478 960 hybrid_clock.cc:648] HybridClock initialized: now 1770152793176467 us; error 28 us; skew 500 ppm
Feb 03 21:06:33 dist-test-slave-j889 krb5kdc[944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:33.178799 960 init.cc:377] Logged in from keytab as kudu/127.0.102.190@KRBTEST.COM (short username kudu)
I20260203 21:06:33.179761 960 webserver.cc:492] Webserver started at http://127.0.102.190:37761/ using document root <none> and password file <none>
I20260203 21:06:33.180282 960 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:33.180325 960 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:33.180472 960 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:33.182108 960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "869d0be3f2ff40d989049ebecba502a9"
format_stamp: "Formatted at 2026-02-03 21:06:33 on dist-test-slave-j889"
server_key: "8c165035f29fc3a923498764f0a54b9a"
server_key_iv: "5cf387325bc5d037fb5c8e37e966b4c4"
server_key_version: "encryptionkey@0"
I20260203 21:06:33.182526 960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "869d0be3f2ff40d989049ebecba502a9"
format_stamp: "Formatted at 2026-02-03 21:06:33 on dist-test-slave-j889"
server_key: "8c165035f29fc3a923498764f0a54b9a"
server_key_iv: "5cf387325bc5d037fb5c8e37e966b4c4"
server_key_version: "encryptionkey@0"
I20260203 21:06:33.185745 960 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260203 21:06:33.187894 975 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:33.188961 960 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:06:33.189064 960 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "869d0be3f2ff40d989049ebecba502a9"
format_stamp: "Formatted at 2026-02-03 21:06:33 on dist-test-slave-j889"
server_key: "8c165035f29fc3a923498764f0a54b9a"
server_key_iv: "5cf387325bc5d037fb5c8e37e966b4c4"
server_key_version: "encryptionkey@0"
I20260203 21:06:33.189144 960 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:33.224681 960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:33.230677 960 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:33.230849 960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:33.238049 960 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:43993
I20260203 21:06:33.238045 1027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:43993 every 8 connection(s)
I20260203 21:06:33.239123 960 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:06:33.241214 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 960
I20260203 21:06:33.241328 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:06:33.241567 410 external_mini_cluster.cc:1468] Setting key a63c7a1fd8b5e9830963ad4eda8f61b0
I20260203 21:06:33.242141 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:33.248441 1028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9: Bootstrap starting.
Feb 03 21:06:33 dist-test-slave-j889 krb5kdc[944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:33.250823 1028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:33.251524 1028 log.cc:826] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:33.253535 1028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9: No bootstrap required, opened a new log
I20260203 21:06:33.254207 1031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:33.242805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48738 (local address 127.0.102.190:43993)
0203 21:06:33.243193 (+ 388us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:33.243202 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:33.243234 (+ 32us) server_negotiation.cc:408] Connection header received
0203 21:06:33.244288 (+ 1054us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:33.244310 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:33.244674 (+ 364us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:33.244958 (+ 284us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:33.245925 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.246807 (+ 882us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:33.247632 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.247828 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:33.250183 (+ 2355us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:33.250224 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:33.250234 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:33.250269 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:33.251907 (+ 1638us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:33.252277 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:33.252282 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:33.252286 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:33.252361 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:33.252585 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:33.252587 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:33.252589 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:33.252938 (+ 349us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:33.253094 (+ 156us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:33.253352 (+ 258us) server_negotiation.cc:300] Negotiation successful
0203 21:06:33.253593 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":118,"threads_started":1}
I20260203 21:06:33.256440 1028 raft_consensus.cc:359] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869d0be3f2ff40d989049ebecba502a9" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43993 } }
I20260203 21:06:33.256659 1028 raft_consensus.cc:385] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:33.256727 1028 raft_consensus.cc:740] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869d0be3f2ff40d989049ebecba502a9, State: Initialized, Role: FOLLOWER
I20260203 21:06:33.257158 1028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869d0be3f2ff40d989049ebecba502a9" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43993 } }
I20260203 21:06:33.257290 1028 raft_consensus.cc:399] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:06:33.257329 1028 raft_consensus.cc:493] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:06:33.257411 1028 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:33.258241 1028 raft_consensus.cc:515] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869d0be3f2ff40d989049ebecba502a9" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43993 } }
I20260203 21:06:33.258569 1028 leader_election.cc:304] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 869d0be3f2ff40d989049ebecba502a9; no voters:
I20260203 21:06:33.258845 1028 leader_election.cc:290] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:06:33.258929 1033 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:33.259070 1033 raft_consensus.cc:697] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [term 1 LEADER]: Becoming Leader. State: Replica: 869d0be3f2ff40d989049ebecba502a9, State: Running, Role: LEADER
I20260203 21:06:33.259328 1033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869d0be3f2ff40d989049ebecba502a9" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43993 } }
I20260203 21:06:33.259981 1028 sys_catalog.cc:565] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:06:33.260972 1034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "869d0be3f2ff40d989049ebecba502a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869d0be3f2ff40d989049ebecba502a9" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43993 } } }
I20260203 21:06:33.260986 1035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869d0be3f2ff40d989049ebecba502a9. Latest consensus state: current_term: 1 leader_uuid: "869d0be3f2ff40d989049ebecba502a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869d0be3f2ff40d989049ebecba502a9" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43993 } } }
I20260203 21:06:33.261089 1034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [sys.catalog]: This master's current role is: LEADER
I20260203 21:06:33.261114 1035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9 [sys.catalog]: This master's current role is: LEADER
I20260203 21:06:33.261487 1040 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:06:33.264382 1040 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:06:33.269222 1040 catalog_manager.cc:1357] Generated new cluster ID: 1a9bfe3e93524205a95ee73f73068fdd
I20260203 21:06:33.269295 1040 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:06:33.286667 1040 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:06:33.287617 1040 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:06:33.298532 1040 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 869d0be3f2ff40d989049ebecba502a9: Generated new TSK 0
I20260203 21:06:33.299223 1040 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:33.347658 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:43993
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:33.452299 1056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:33.452513 1056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:33.452586 1056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:33.455803 1056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:33.455869 1056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:33.455960 1056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:06:33.460161 1056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:43993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:33.461237 1056 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:33.462142 1056 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:33.468679 1061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:33.468680 1062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:33.468679 1064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:33.468920 1056 server_base.cc:1060] running on GCE node
I20260203 21:06:33.469235 1056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:33.469774 1056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:33.470935 1056 hybrid_clock.cc:648] HybridClock initialized: now 1770152793470921 us; error 30 us; skew 500 ppm
Feb 03 21:06:33 dist-test-slave-j889 krb5kdc[944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:33.473644 1056 init.cc:377] Logged in from keytab as kudu/127.0.102.129@KRBTEST.COM (short username kudu)
I20260203 21:06:33.474715 1056 webserver.cc:492] Webserver started at http://127.0.102.129:45777/ using document root <none> and password file <none>
I20260203 21:06:33.475260 1056 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:33.475319 1056 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:33.475494 1056 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:33.477322 1056 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "4d33c17d7a5a4e38b2b5ea52205c043d"
format_stamp: "Formatted at 2026-02-03 21:06:33 on dist-test-slave-j889"
server_key: "c1cba72e31436f4a56041254b5f05baa"
server_key_iv: "b10deac9244d3b64d98d0b88a685038d"
server_key_version: "encryptionkey@0"
I20260203 21:06:33.477784 1056 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "4d33c17d7a5a4e38b2b5ea52205c043d"
format_stamp: "Formatted at 2026-02-03 21:06:33 on dist-test-slave-j889"
server_key: "c1cba72e31436f4a56041254b5f05baa"
server_key_iv: "b10deac9244d3b64d98d0b88a685038d"
server_key_version: "encryptionkey@0"
I20260203 21:06:33.481106 1056 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260203 21:06:33.483382 1071 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:33.484444 1056 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:06:33.484548 1056 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "4d33c17d7a5a4e38b2b5ea52205c043d"
format_stamp: "Formatted at 2026-02-03 21:06:33 on dist-test-slave-j889"
server_key: "c1cba72e31436f4a56041254b5f05baa"
server_key_iv: "b10deac9244d3b64d98d0b88a685038d"
server_key_version: "encryptionkey@0"
I20260203 21:06:33.484660 1056 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:33.511454 1056 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:33.514370 1056 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:33.514544 1056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:33.515089 1056 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:33.515974 1056 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:33.516021 1056 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:33.516073 1056 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:33.516099 1056 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:33.524689 1056 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:43275
I20260203 21:06:33.524713 1184 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:43275 every 8 connection(s)
I20260203 21:06:33.525594 1056 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:06:33.532929 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 1056
I20260203 21:06:33.533027 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:06:33.533233 410 external_mini_cluster.cc:1468] Setting key ebe18d041b6945607c2e387e9fda7180
Feb 03 21:06:33 dist-test-slave-j889 krb5kdc[944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:33.538584 1031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:33.527347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:36265 (local address 127.0.102.190:43993)
0203 21:06:33.527488 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:33.527493 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:33.528151 (+ 658us) server_negotiation.cc:408] Connection header received
0203 21:06:33.529184 (+ 1033us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:33.529187 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:33.529242 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:33.529324 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:33.531061 (+ 1737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.531638 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:33.532274 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.532425 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:33.534732 (+ 2307us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:33.534750 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:33.534754 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:33.534785 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:33.536430 (+ 1645us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:33.537064 (+ 634us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:33.537067 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:33.537068 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:33.537118 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:33.537459 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:33.537462 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:33.537464 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:33.537640 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:33.537719 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:33.538339 (+ 620us) server_negotiation.cc:300] Negotiation successful
0203 21:06:33.538449 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260203 21:06:33.539475 1187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:33.527602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:43993 (local address 127.0.102.129:36265)
0203 21:06:33.528016 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:33.528052 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:33.528952 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:33.529519 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:33.529528 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:33.529940 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:33.530883 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:33.530898 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.531755 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:33.531758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:33.532176 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:33.532183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.532353 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:33.532975 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:33.533011 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:33.534578 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:33.536587 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:33.536594 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:33.536608 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:33.536963 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:33.537198 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:33.537201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:33.537203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:33.537332 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:33.537734 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:33.537741 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:33.538050 (+ 309us) client_negotiation.cc:769] Sending connection context
0203 21:06:33.538304 (+ 254us) client_negotiation.cc:240] Negotiation successful
0203 21:06:33.538602 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":117,"threads_started":1}
I20260203 21:06:33.540706 1185 heartbeater.cc:344] Connected to a master server at 127.0.102.190:43993
I20260203 21:06:33.540943 1185 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:33.541451 1185 heartbeater.cc:507] Master 127.0.102.190:43993 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.102.130@KRBTEST.COM; defaulting to no policy
I20260203 21:06:33.542872 992 ts_manager.cc:194] Registered new tserver with Master: 4d33c17d7a5a4e38b2b5ea52205c043d (127.0.102.129:43275)
I20260203 21:06:33.544255 992 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:36265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:33.576455 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:43993
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:33.679316 1192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:33.679518 1192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:33.679600 1192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:33.682796 1192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:33.682853 1192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:33.682952 1192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:06:33.686991 1192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:43993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:33.687997 1192 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:33.688854 1192 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:33.694864 1198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:33.694875 1197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:33.694975 1192 server_base.cc:1060] running on GCE node
W20260203 21:06:33.694870 1200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:33.695417 1192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:33.695914 1192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:33.697049 1192 hybrid_clock.cc:648] HybridClock initialized: now 1770152793697036 us; error 26 us; skew 500 ppm
Feb 03 21:06:33 dist-test-slave-j889 krb5kdc[944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:33.699350 1192 init.cc:377] Logged in from keytab as kudu/127.0.102.130@KRBTEST.COM (short username kudu)
I20260203 21:06:33.700314 1192 webserver.cc:492] Webserver started at http://127.0.102.130:44475/ using document root <none> and password file <none>
I20260203 21:06:33.700863 1192 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:33.700912 1192 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:33.701076 1192 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:33.702644 1192 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "8131650bf27b41a2996316489e576fa7"
format_stamp: "Formatted at 2026-02-03 21:06:33 on dist-test-slave-j889"
server_key: "753a7bf02f505659fa8c3dbcf03438e6"
server_key_iv: "675a37cfafe0cf01ff299b5a4641a563"
server_key_version: "encryptionkey@0"
I20260203 21:06:33.703078 1192 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "8131650bf27b41a2996316489e576fa7"
format_stamp: "Formatted at 2026-02-03 21:06:33 on dist-test-slave-j889"
server_key: "753a7bf02f505659fa8c3dbcf03438e6"
server_key_iv: "675a37cfafe0cf01ff299b5a4641a563"
server_key_version: "encryptionkey@0"
I20260203 21:06:33.706224 1192 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260203 21:06:33.708211 1207 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:33.709205 1192 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:06:33.709304 1192 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "8131650bf27b41a2996316489e576fa7"
format_stamp: "Formatted at 2026-02-03 21:06:33 on dist-test-slave-j889"
server_key: "753a7bf02f505659fa8c3dbcf03438e6"
server_key_iv: "675a37cfafe0cf01ff299b5a4641a563"
server_key_version: "encryptionkey@0"
I20260203 21:06:33.709381 1192 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:33.723568 1192 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:33.726215 1192 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:33.726399 1192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:33.726935 1192 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:33.727761 1192 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:33.727806 1192 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:33.727856 1192 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:33.727880 1192 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:33.736541 1192 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:39143
I20260203 21:06:33.736608 1320 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:39143 every 8 connection(s)
I20260203 21:06:33.737473 1192 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:06:33.741461 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 1192
I20260203 21:06:33.741590 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:06:33.741859 410 external_mini_cluster.cc:1468] Setting key 5f1051da057a7c73d0a61796da1e12cc
Feb 03 21:06:33 dist-test-slave-j889 krb5kdc[944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:33.749922 1031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:33.739095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:39609 (local address 127.0.102.190:43993)
0203 21:06:33.739227 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:33.739231 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:33.739917 (+ 686us) server_negotiation.cc:408] Connection header received
0203 21:06:33.740737 (+ 820us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:33.740741 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:33.740789 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:33.740860 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:33.742881 (+ 2021us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.743492 (+ 611us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:33.744362 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.744504 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:33.746696 (+ 2192us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:33.746713 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:33.746716 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:33.746742 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:33.748085 (+ 1343us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:33.748551 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:33.748554 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:33.748556 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:33.748612 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:33.748897 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:33.748900 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:33.748902 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:33.749047 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:33.749130 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:33.749643 (+ 513us) server_negotiation.cc:300] Negotiation successful
0203 21:06:33.749761 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.0.102.131@KRBTEST.COM; defaulting to no policy
I20260203 21:06:33.750574 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:33.739330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:43993 (local address 127.0.102.130:39609)
0203 21:06:33.739769 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:33.739806 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:33.740530 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:33.740997 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:33.741005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:33.741513 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:33.742720 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:33.742736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.743632 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:33.743637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:33.744199 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:33.744211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.744432 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:33.745116 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:33.745142 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:33.746554 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:33.748203 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:33.748208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:33.748220 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:33.748458 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:33.748716 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:33.748720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:33.748722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:33.748808 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:33.749139 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:33.749146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:33.749400 (+ 254us) client_negotiation.cc:769] Sending connection context
0203 21:06:33.749587 (+ 187us) client_negotiation.cc:240] Negotiation successful
0203 21:06:33.749853 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":104,"threads_started":1}
I20260203 21:06:33.751675 1321 heartbeater.cc:344] Connected to a master server at 127.0.102.190:43993
I20260203 21:06:33.751905 1321 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.131@KRBTEST.COM" created.
I20260203 21:06:33.752498 1321 heartbeater.cc:507] Master 127.0.102.190:43993 requested a full tablet report, sending...
I20260203 21:06:33.753610 992 ts_manager.cc:194] Registered new tserver with Master: 8131650bf27b41a2996316489e576fa7 (127.0.102.130:39143)
I20260203 21:06:33.754343 992 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.130@KRBTEST.COM'} at 127.0.102.130:39609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:33.783214 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:43993
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:38161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:33.884691 1328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:33.884893 1328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:33.884968 1328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:33.888106 1328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:33.888166 1328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:33.888264 1328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:06:33.892377 1328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:38161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:43993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:33.893395 1328 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:33.894217 1328 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:33.900161 1333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:33.900183 1336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:33.900177 1334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:33.900342 1328 server_base.cc:1060] running on GCE node
I20260203 21:06:33.900789 1328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:33.901265 1328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:33.902407 1328 hybrid_clock.cc:648] HybridClock initialized: now 1770152793902393 us; error 31 us; skew 500 ppm
Feb 03 21:06:33 dist-test-slave-j889 krb5kdc[944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:33.904804 1328 init.cc:377] Logged in from keytab as kudu/127.0.102.131@KRBTEST.COM (short username kudu)
I20260203 21:06:33.905701 1328 webserver.cc:492] Webserver started at http://127.0.102.131:42043/ using document root <none> and password file <none>
I20260203 21:06:33.906184 1328 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:33.906230 1328 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:33.906388 1328 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:33.907990 1328 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "88bf94963b2349c685278c504f7b6548"
format_stamp: "Formatted at 2026-02-03 21:06:33 on dist-test-slave-j889"
server_key: "965857dac3da3adaae922b9cf21b080b"
server_key_iv: "fcf5044f02479f772db30c8d348428b0"
server_key_version: "encryptionkey@0"
I20260203 21:06:33.908416 1328 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "88bf94963b2349c685278c504f7b6548"
format_stamp: "Formatted at 2026-02-03 21:06:33 on dist-test-slave-j889"
server_key: "965857dac3da3adaae922b9cf21b080b"
server_key_iv: "fcf5044f02479f772db30c8d348428b0"
server_key_version: "encryptionkey@0"
I20260203 21:06:33.911540 1328 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260203 21:06:33.913559 1343 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:33.914531 1328 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260203 21:06:33.914633 1328 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "88bf94963b2349c685278c504f7b6548"
format_stamp: "Formatted at 2026-02-03 21:06:33 on dist-test-slave-j889"
server_key: "965857dac3da3adaae922b9cf21b080b"
server_key_iv: "fcf5044f02479f772db30c8d348428b0"
server_key_version: "encryptionkey@0"
I20260203 21:06:33.914716 1328 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:33.934762 1328 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:33.937357 1328 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:33.937525 1328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:33.938089 1328 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:33.938941 1328 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:33.938987 1328 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:33.939025 1328 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:33.939054 1328 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:33.947674 1328 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:46133
I20260203 21:06:33.947710 1456 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:46133 every 8 connection(s)
I20260203 21:06:33.948715 1328 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
Feb 03 21:06:33 dist-test-slave-j889 krb5kdc[944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:33.958235 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 1328
I20260203 21:06:33.958349 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:06:33.958590 410 external_mini_cluster.cc:1468] Setting key bc727df0e9f010f084b801b6d8312221
I20260203 21:06:33.960494 1031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:33.950454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:60895 (local address 127.0.102.190:43993)
0203 21:06:33.950619 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:33.950623 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:33.951281 (+ 658us) server_negotiation.cc:408] Connection header received
0203 21:06:33.952083 (+ 802us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:33.952086 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:33.952134 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:33.952206 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:33.953849 (+ 1643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.954339 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:33.954985 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.955111 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:33.957258 (+ 2147us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:33.957277 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:33.957279 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:33.957303 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:33.958612 (+ 1309us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:33.959063 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:33.959067 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:33.959069 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:33.959114 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:33.959408 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:33.959411 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:33.959413 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:33.959591 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:33.959692 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:33.960222 (+ 530us) server_negotiation.cc:300] Negotiation successful
0203 21:06:33.960338 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260203 21:06:33.961144 1459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:33.950705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:43993 (local address 127.0.102.131:60895)
0203 21:06:33.951123 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:33.951160 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:33.951909 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:33.952368 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:33.952377 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:33.952803 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:33.953726 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:33.953740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.954478 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:33.954482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:33.954897 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:33.954904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.955073 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:33.955651 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:33.955678 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:33.957111 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:33.958719 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:33.958724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:33.958733 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:33.958962 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:33.959203 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:33.959206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:33.959208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:33.959311 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:33.959699 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:33.959705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:33.959990 (+ 285us) client_negotiation.cc:769] Sending connection context
0203 21:06:33.960174 (+ 184us) client_negotiation.cc:240] Negotiation successful
0203 21:06:33.960452 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":110,"threads_started":1}
I20260203 21:06:33.962108 1457 heartbeater.cc:344] Connected to a master server at 127.0.102.190:43993
I20260203 21:06:33.962330 1457 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:33.962845 1457 heartbeater.cc:507] Master 127.0.102.190:43993 requested a full tablet report, sending...
I20260203 21:06:33.963735 992 ts_manager.cc:194] Registered new tserver with Master: 88bf94963b2349c685278c504f7b6548 (127.0.102.131:46133)
I20260203 21:06:33.964273 992 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.131@KRBTEST.COM'} at 127.0.102.131:60895
I20260203 21:06:33.971984 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260203 21:06:33.980180 1031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:33.974602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48740 (local address 127.0.102.190:43993)
0203 21:06:33.974727 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:33.974731 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:33.974862 (+ 131us) server_negotiation.cc:408] Connection header received
0203 21:06:33.974952 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:33.974955 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:33.974993 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:33.975055 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:33.975753 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.976230 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:33.976864 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.976992 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:33.977899 (+ 907us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:33.977912 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:33.977914 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:33.977936 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:33.979182 (+ 1246us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:33.979460 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:33.979462 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:33.979463 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:33.979497 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:33.979627 (+ 130us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:33.979628 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:33.979630 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:33.979763 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:33.979854 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:33.979991 (+ 137us) server_negotiation.cc:300] Negotiation successful
0203 21:06:33.980074 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260203 21:06:33.983816 992 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260203 21:06:33.986107 992 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:06:33.997612 1468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:33.993400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:39143 (local address 127.0.0.1:48126)
0203 21:06:33.993972 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:33.994006 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:33.994138 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:33.994837 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:33.994843 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:33.994864 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:33.995141 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:33.995149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.996396 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:33.996399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:33.997162 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:33.997169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.997266 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:33.997312 (+ 46us) client_negotiation.cc:769] Sending connection context
0203 21:06:33.997401 (+ 89us) client_negotiation.cc:240] Negotiation successful
0203 21:06:33.997483 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":70,"threads_started":1}
I20260203 21:06:33.998092 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:33.993581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:43275 (local address 127.0.0.1:43424)
0203 21:06:33.994347 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:33.994359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:33.994576 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:33.995265 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:33.995268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:33.995288 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:33.995558 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:33.995565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.996920 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:33.996924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:33.997769 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:33.997778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.997873 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:33.997888 (+ 15us) client_negotiation.cc:769] Sending connection context
0203 21:06:33.997935 (+ 47us) client_negotiation.cc:240] Negotiation successful
0203 21:06:33.997987 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":82,"threads_started":1}
I20260203 21:06:33.998417 1471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:33.994069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:46133 (local address 127.0.0.1:34468)
0203 21:06:33.995350 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:33.995363 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:33.995444 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:33.995815 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:33.995818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:33.995836 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:33.996076 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:33.996082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.997275 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:33.997278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:33.998063 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:33.998072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.998170 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:33.998186 (+ 16us) client_negotiation.cc:769] Sending connection context
0203 21:06:33.998228 (+ 42us) client_negotiation.cc:240] Negotiation successful
0203 21:06:33.998280 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1222,"thread_start_us":79,"threads_started":1}
I20260203 21:06:33.998826 1470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:33.993586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48126 (local address 127.0.102.130:39143)
0203 21:06:33.993956 (+ 370us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:33.993963 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:33.994256 (+ 293us) server_negotiation.cc:408] Connection header received
0203 21:06:33.994342 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:33.994347 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:33.994524 (+ 177us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:33.994673 (+ 149us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:33.995262 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.996296 (+ 1034us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:33.997958 (+ 1662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.998451 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:33.998558 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:33.998615 (+ 57us) server_negotiation.cc:300] Negotiation successful
0203 21:06:33.998704 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":99,"threads_started":1}
I20260203 21:06:33.999130 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:33.994242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.0.102.129:43275)
0203 21:06:33.994903 (+ 661us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:33.994909 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:33.994928 (+ 19us) server_negotiation.cc:408] Connection header received
0203 21:06:33.994999 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:33.995004 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:33.995140 (+ 136us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:33.995264 (+ 124us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:33.995674 (+ 410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.996699 (+ 1025us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:33.998337 (+ 1638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.998813 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:33.998909 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:33.998965 (+ 56us) server_negotiation.cc:300] Negotiation successful
0203 21:06:33.999034 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":103,"threads_started":1}
I20260203 21:06:33.999202 1473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:33.994535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.0.102.131:46133)
0203 21:06:33.994847 (+ 312us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:33.994852 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:33.995423 (+ 571us) server_negotiation.cc:408] Connection header received
0203 21:06:33.995554 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:33.995558 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:33.995689 (+ 131us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:33.995818 (+ 129us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:33.996174 (+ 356us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.997163 (+ 989us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:33.998311 (+ 1148us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:33.998852 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:33.998952 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:33.999010 (+ 58us) server_negotiation.cc:300] Negotiation successful
0203 21:06:33.999083 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":128,"threads_started":1}
I20260203 21:06:34.001421 1391 tablet_service.cc:1505] Processing CreateTablet for tablet 123e40fcedf44e5696867a05b929b04b (DEFAULT_TABLE table=test-table [id=9a7b7413b5da40f0860f0b802a0f4ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:34.001515 1119 tablet_service.cc:1505] Processing CreateTablet for tablet 123e40fcedf44e5696867a05b929b04b (DEFAULT_TABLE table=test-table [id=9a7b7413b5da40f0860f0b802a0f4ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:34.001515 1255 tablet_service.cc:1505] Processing CreateTablet for tablet 123e40fcedf44e5696867a05b929b04b (DEFAULT_TABLE table=test-table [id=9a7b7413b5da40f0860f0b802a0f4ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:34.002233 1391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 123e40fcedf44e5696867a05b929b04b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:34.002341 1119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 123e40fcedf44e5696867a05b929b04b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:34.002341 1255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 123e40fcedf44e5696867a05b929b04b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:34.007072 1474 tablet_bootstrap.cc:492] T 123e40fcedf44e5696867a05b929b04b P 88bf94963b2349c685278c504f7b6548: Bootstrap starting.
I20260203 21:06:34.008364 1475 tablet_bootstrap.cc:492] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7: Bootstrap starting.
I20260203 21:06:34.008910 1476 tablet_bootstrap.cc:492] T 123e40fcedf44e5696867a05b929b04b P 4d33c17d7a5a4e38b2b5ea52205c043d: Bootstrap starting.
I20260203 21:06:34.009485 1474 tablet_bootstrap.cc:654] T 123e40fcedf44e5696867a05b929b04b P 88bf94963b2349c685278c504f7b6548: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:34.010119 1475 tablet_bootstrap.cc:654] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:34.010274 1474 log.cc:826] T 123e40fcedf44e5696867a05b929b04b P 88bf94963b2349c685278c504f7b6548: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:34.010736 1476 tablet_bootstrap.cc:654] T 123e40fcedf44e5696867a05b929b04b P 4d33c17d7a5a4e38b2b5ea52205c043d: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:34.010790 1475 log.cc:826] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:34.011452 1476 log.cc:826] T 123e40fcedf44e5696867a05b929b04b P 4d33c17d7a5a4e38b2b5ea52205c043d: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:34.012022 1474 tablet_bootstrap.cc:492] T 123e40fcedf44e5696867a05b929b04b P 88bf94963b2349c685278c504f7b6548: No bootstrap required, opened a new log
I20260203 21:06:34.012194 1474 ts_tablet_manager.cc:1403] T 123e40fcedf44e5696867a05b929b04b P 88bf94963b2349c685278c504f7b6548: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260203 21:06:34.012287 1475 tablet_bootstrap.cc:492] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7: No bootstrap required, opened a new log
I20260203 21:06:34.012451 1475 ts_tablet_manager.cc:1403] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260203 21:06:34.013080 1476 tablet_bootstrap.cc:492] T 123e40fcedf44e5696867a05b929b04b P 4d33c17d7a5a4e38b2b5ea52205c043d: No bootstrap required, opened a new log
I20260203 21:06:34.013233 1476 ts_tablet_manager.cc:1403] T 123e40fcedf44e5696867a05b929b04b P 4d33c17d7a5a4e38b2b5ea52205c043d: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260203 21:06:34.015714 1475 raft_consensus.cc:359] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8131650bf27b41a2996316489e576fa7" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 39143 } } peers { permanent_uuid: "4d33c17d7a5a4e38b2b5ea52205c043d" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 43275 } } peers { permanent_uuid: "88bf94963b2349c685278c504f7b6548" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46133 } }
I20260203 21:06:34.015888 1475 raft_consensus.cc:385] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:34.015939 1475 raft_consensus.cc:740] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8131650bf27b41a2996316489e576fa7, State: Initialized, Role: FOLLOWER
I20260203 21:06:34.016141 1474 raft_consensus.cc:359] T 123e40fcedf44e5696867a05b929b04b P 88bf94963b2349c685278c504f7b6548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8131650bf27b41a2996316489e576fa7" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 39143 } } peers { permanent_uuid: "4d33c17d7a5a4e38b2b5ea52205c043d" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 43275 } } peers { permanent_uuid: "88bf94963b2349c685278c504f7b6548" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46133 } }
I20260203 21:06:34.016335 1474 raft_consensus.cc:385] T 123e40fcedf44e5696867a05b929b04b P 88bf94963b2349c685278c504f7b6548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:34.016321 1475 consensus_queue.cc:260] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8131650bf27b41a2996316489e576fa7" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 39143 } } peers { permanent_uuid: "4d33c17d7a5a4e38b2b5ea52205c043d" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 43275 } } peers { permanent_uuid: "88bf94963b2349c685278c504f7b6548" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46133 } }
I20260203 21:06:34.016403 1474 raft_consensus.cc:740] T 123e40fcedf44e5696867a05b929b04b P 88bf94963b2349c685278c504f7b6548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88bf94963b2349c685278c504f7b6548, State: Initialized, Role: FOLLOWER
I20260203 21:06:34.016579 1476 raft_consensus.cc:359] T 123e40fcedf44e5696867a05b929b04b P 4d33c17d7a5a4e38b2b5ea52205c043d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8131650bf27b41a2996316489e576fa7" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 39143 } } peers { permanent_uuid: "4d33c17d7a5a4e38b2b5ea52205c043d" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 43275 } } peers { permanent_uuid: "88bf94963b2349c685278c504f7b6548" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46133 } }
I20260203 21:06:34.016789 1476 raft_consensus.cc:385] T 123e40fcedf44e5696867a05b929b04b P 4d33c17d7a5a4e38b2b5ea52205c043d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:34.016850 1476 raft_consensus.cc:740] T 123e40fcedf44e5696867a05b929b04b P 4d33c17d7a5a4e38b2b5ea52205c043d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d33c17d7a5a4e38b2b5ea52205c043d, State: Initialized, Role: FOLLOWER
I20260203 21:06:34.016861 1474 consensus_queue.cc:260] T 123e40fcedf44e5696867a05b929b04b P 88bf94963b2349c685278c504f7b6548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8131650bf27b41a2996316489e576fa7" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 39143 } } peers { permanent_uuid: "4d33c17d7a5a4e38b2b5ea52205c043d" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 43275 } } peers { permanent_uuid: "88bf94963b2349c685278c504f7b6548" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46133 } }
I20260203 21:06:34.017154 1321 heartbeater.cc:499] Master 127.0.102.190:43993 was elected leader, sending a full tablet report...
I20260203 21:06:34.017238 1475 ts_tablet_manager.cc:1434] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260203 21:06:34.017230 1476 consensus_queue.cc:260] T 123e40fcedf44e5696867a05b929b04b P 4d33c17d7a5a4e38b2b5ea52205c043d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8131650bf27b41a2996316489e576fa7" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 39143 } } peers { permanent_uuid: "4d33c17d7a5a4e38b2b5ea52205c043d" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 43275 } } peers { permanent_uuid: "88bf94963b2349c685278c504f7b6548" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46133 } }
I20260203 21:06:34.017696 1474 ts_tablet_manager.cc:1434] T 123e40fcedf44e5696867a05b929b04b P 88bf94963b2349c685278c504f7b6548: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260203 21:06:34.017822 1185 heartbeater.cc:499] Master 127.0.102.190:43993 was elected leader, sending a full tablet report...
I20260203 21:06:34.017949 1476 ts_tablet_manager.cc:1434] T 123e40fcedf44e5696867a05b929b04b P 4d33c17d7a5a4e38b2b5ea52205c043d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260203 21:06:34.018003 1457 heartbeater.cc:499] Master 127.0.102.190:43993 was elected leader, sending a full tablet report...
W20260203 21:06:34.026628 1186 tablet.cc:2378] T 123e40fcedf44e5696867a05b929b04b P 4d33c17d7a5a4e38b2b5ea52205c043d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260203 21:06:34.161640 1480 raft_consensus.cc:493] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:06:34.161837 1480 raft_consensus.cc:515] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8131650bf27b41a2996316489e576fa7" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 39143 } } peers { permanent_uuid: "4d33c17d7a5a4e38b2b5ea52205c043d" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 43275 } } peers { permanent_uuid: "88bf94963b2349c685278c504f7b6548" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46133 } }
I20260203 21:06:34.163048 1480 leader_election.cc:290] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d33c17d7a5a4e38b2b5ea52205c043d (127.0.102.129:43275), 88bf94963b2349c685278c504f7b6548 (127.0.102.131:46133)
I20260203 21:06:34.166124 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:34.163280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:46133 (local address 127.0.102.130:54947)
0203 21:06:34.163392 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:34.163407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:34.163499 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:34.163757 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:34.163760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:34.163786 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:34.164041 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:34.164048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:34.164989 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:34.164992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:34.165754 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:34.165763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:34.165872 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:34.165893 (+ 21us) client_negotiation.cc:769] Sending connection context
0203 21:06:34.165939 (+ 46us) client_negotiation.cc:240] Negotiation successful
0203 21:06:34.165995 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260203 21:06:34.166512 1473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:34.163380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:54947 (local address 127.0.102.131:46133)
0203 21:06:34.163531 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:34.163536 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:34.163552 (+ 16us) server_negotiation.cc:408] Connection header received
0203 21:06:34.163601 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:34.163605 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:34.163653 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:34.163728 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:34.164158 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:34.164881 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:34.165877 (+ 996us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:34.166244 (+ 367us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:34.166286 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:34.166331 (+ 45us) server_negotiation.cc:300] Negotiation successful
0203 21:06:34.166413 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260203 21:06:34.167012 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:34.163280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:43275 (local address 127.0.102.130:48731)
0203 21:06:34.163920 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:34.163934 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:34.164029 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:34.164256 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:34.164258 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:34.164273 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:34.164488 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:34.164493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:34.165303 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:34.165306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:34.166142 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:34.166151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:34.166768 (+ 617us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:34.166785 (+ 17us) client_negotiation.cc:769] Sending connection context
0203 21:06:34.166857 (+ 72us) client_negotiation.cc:240] Negotiation successful
0203 21:06:34.166908 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":294,"thread_start_us":164,"threads_started":1}
I20260203 21:06:34.167098 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:34.163380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:48731 (local address 127.0.102.129:43275)
0203 21:06:34.163505 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:34.163508 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:34.164044 (+ 536us) server_negotiation.cc:408] Connection header received
0203 21:06:34.164101 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:34.164104 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:34.164157 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:34.164244 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:34.164596 (+ 352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:34.165208 (+ 612us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:34.166283 (+ 1075us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:34.166671 (+ 388us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:34.166707 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:34.166908 (+ 201us) server_negotiation.cc:300] Negotiation successful
0203 21:06:34.166992 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260203 21:06:34.167124 1411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "123e40fcedf44e5696867a05b929b04b" candidate_uuid: "8131650bf27b41a2996316489e576fa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88bf94963b2349c685278c504f7b6548" is_pre_election: true
I20260203 21:06:34.167419 1411 raft_consensus.cc:2468] T 123e40fcedf44e5696867a05b929b04b P 88bf94963b2349c685278c504f7b6548 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8131650bf27b41a2996316489e576fa7 in term 0.
I20260203 21:06:34.167645 1139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "123e40fcedf44e5696867a05b929b04b" candidate_uuid: "8131650bf27b41a2996316489e576fa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d33c17d7a5a4e38b2b5ea52205c043d" is_pre_election: true
I20260203 21:06:34.167904 1139 raft_consensus.cc:2468] T 123e40fcedf44e5696867a05b929b04b P 4d33c17d7a5a4e38b2b5ea52205c043d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8131650bf27b41a2996316489e576fa7 in term 0.
I20260203 21:06:34.167892 1211 leader_election.cc:304] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8131650bf27b41a2996316489e576fa7, 88bf94963b2349c685278c504f7b6548; no voters:
I20260203 21:06:34.168109 1480 raft_consensus.cc:2804] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:06:34.168185 1480 raft_consensus.cc:493] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:06:34.168233 1480 raft_consensus.cc:3060] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:34.169178 1480 raft_consensus.cc:515] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8131650bf27b41a2996316489e576fa7" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 39143 } } peers { permanent_uuid: "4d33c17d7a5a4e38b2b5ea52205c043d" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 43275 } } peers { permanent_uuid: "88bf94963b2349c685278c504f7b6548" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46133 } }
I20260203 21:06:34.169507 1480 leader_election.cc:290] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [CANDIDATE]: Term 1 election: Requested vote from peers 4d33c17d7a5a4e38b2b5ea52205c043d (127.0.102.129:43275), 88bf94963b2349c685278c504f7b6548 (127.0.102.131:46133)
I20260203 21:06:34.169888 1411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "123e40fcedf44e5696867a05b929b04b" candidate_uuid: "8131650bf27b41a2996316489e576fa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88bf94963b2349c685278c504f7b6548"
I20260203 21:06:34.169963 1139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "123e40fcedf44e5696867a05b929b04b" candidate_uuid: "8131650bf27b41a2996316489e576fa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d33c17d7a5a4e38b2b5ea52205c043d"
I20260203 21:06:34.170018 1411 raft_consensus.cc:3060] T 123e40fcedf44e5696867a05b929b04b P 88bf94963b2349c685278c504f7b6548 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:34.170068 1139 raft_consensus.cc:3060] T 123e40fcedf44e5696867a05b929b04b P 4d33c17d7a5a4e38b2b5ea52205c043d [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:34.170912 1139 raft_consensus.cc:2468] T 123e40fcedf44e5696867a05b929b04b P 4d33c17d7a5a4e38b2b5ea52205c043d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8131650bf27b41a2996316489e576fa7 in term 1.
I20260203 21:06:34.170998 1411 raft_consensus.cc:2468] T 123e40fcedf44e5696867a05b929b04b P 88bf94963b2349c685278c504f7b6548 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8131650bf27b41a2996316489e576fa7 in term 1.
I20260203 21:06:34.171738 1210 leader_election.cc:304] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d33c17d7a5a4e38b2b5ea52205c043d, 8131650bf27b41a2996316489e576fa7; no voters:
I20260203 21:06:34.171916 1480 raft_consensus.cc:2804] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:34.172102 1480 raft_consensus.cc:697] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [term 1 LEADER]: Becoming Leader. State: Replica: 8131650bf27b41a2996316489e576fa7, State: Running, Role: LEADER
I20260203 21:06:34.172405 1480 consensus_queue.cc:237] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8131650bf27b41a2996316489e576fa7" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 39143 } } peers { permanent_uuid: "4d33c17d7a5a4e38b2b5ea52205c043d" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 43275 } } peers { permanent_uuid: "88bf94963b2349c685278c504f7b6548" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46133 } }
I20260203 21:06:34.175324 992 catalog_manager.cc:5654] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8131650bf27b41a2996316489e576fa7 (127.0.102.130). New cstate: current_term: 1 leader_uuid: "8131650bf27b41a2996316489e576fa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8131650bf27b41a2996316489e576fa7" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 39143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d33c17d7a5a4e38b2b5ea52205c043d" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 43275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88bf94963b2349c685278c504f7b6548" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46133 } health_report { overall_health: UNKNOWN } } }
Feb 03 21:06:34 dist-test-slave-j889 krb5kdc[944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.129@KRBTEST.COM
I20260203 21:06:34.191522 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:34.183670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.0.102.129:43275)
0203 21:06:34.183799 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:34.183803 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:34.183896 (+ 93us) server_negotiation.cc:408] Connection header received
0203 21:06:34.184022 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:34.184024 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:34.184065 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:34.184120 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:34.184921 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:34.185369 (+ 448us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:34.185993 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:34.186113 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:34.188269 (+ 2156us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:34.188293 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:34.188305 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:34.188332 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:34.190049 (+ 1717us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:34.190437 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:34.190441 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:34.190443 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:34.190502 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:34.190671 (+ 169us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:34.190673 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:34.190675 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:34.190858 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:34.191011 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:34.191228 (+ 217us) server_negotiation.cc:300] Negotiation successful
0203 21:06:34.191367 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260203 21:06:34.199554 1458 tablet.cc:2378] T 123e40fcedf44e5696867a05b929b04b P 88bf94963b2349c685278c504f7b6548: Can't schedule compaction. Clean time has not been advanced past its initial value.
Feb 03 21:06:34 dist-test-slave-j889 krb5kdc[944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152794, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 03 21:06:34 dist-test-slave-j889 krb5kdc[944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152794, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.102.129@KRBTEST.COM
I20260203 21:06:34.214596 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:34.207539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.0.102.129:43275)
0203 21:06:34.207729 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:34.207734 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:34.207838 (+ 104us) server_negotiation.cc:408] Connection header received
0203 21:06:34.207955 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:34.207958 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:34.208001 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:34.208064 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:34.208835 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:34.209279 (+ 444us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:34.209884 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:34.210075 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:34.211965 (+ 1890us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:34.211980 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:34.211982 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:34.212005 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:34.213353 (+ 1348us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:34.213691 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:34.213694 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:34.213696 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:34.213735 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:34.213915 (+ 180us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:34.213918 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:34.213920 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:34.214039 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:34.214150 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:34.214347 (+ 197us) server_negotiation.cc:300] Negotiation successful
0203 21:06:34.214445 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Feb 03 21:06:34 dist-test-slave-j889 krb5kdc[944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 03 21:06:34 dist-test-slave-j889 krb5kdc[944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.129@KRBTEST.COM
I20260203 21:06:34.236059 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:34.229420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.0.102.129:43275)
0203 21:06:34.229569 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:34.229573 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:34.229634 (+ 61us) server_negotiation.cc:408] Connection header received
0203 21:06:34.229753 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:34.229757 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:34.229801 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:34.229868 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:34.230602 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:34.231064 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:34.231683 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:34.231872 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:34.233828 (+ 1956us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:34.233844 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:34.233847 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:34.233871 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:34.234971 (+ 1100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:34.235288 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:34.235291 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:34.235293 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:34.235326 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:34.235485 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:34.235487 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:34.235488 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:34.235604 (+ 116us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:34.235691 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:34.235856 (+ 165us) server_negotiation.cc:300] Negotiation successful
0203 21:06:34.235938 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260203 21:06:34.237329 1099 tablet_service.cc:3012] Rejecting scan request for tablet 123e40fcedf44e5696867a05b929b04b: Uninitialized: clean time has not yet been initialized
W20260203 21:06:34.238447 1322 tablet.cc:2378] T 123e40fcedf44e5696867a05b929b04b P 8131650bf27b41a2996316489e576fa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260203 21:06:34.239533 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 1056
I20260203 21:06:34.244947 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 1192
I20260203 21:06:34.250782 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 1328
I20260203 21:06:34.256407 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 960
2026-02-03T21:06:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3174 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:06:34 dist-test-slave-j889 krb5kdc[1498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:06:34 dist-test-slave-j889 krb5kdc[1498](info): set up 2 sockets
Feb 03 21:06:34 dist-test-slave-j889 krb5kdc[1498](info): commencing operation
krb5kdc: starting...
W20260203 21:06:36.169044 410 mini_kdc.cc:121] Time spent starting KDC: real 1.892s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:06:36 dist-test-slave-j889 krb5kdc[1498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:06:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:06:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:36.334162 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:37333
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:46633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:37333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:36.457098 1514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:36.457387 1514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:36.457461 1514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:36.461521 1514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:06:36.461603 1514 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260203 21:06:36.461621 1514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:36.461643 1514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:36.461660 1514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:06:36.461683 1514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:06:36.466665 1514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:37333
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:37333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:36.468014 1514 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:36.468962 1514 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:36.475498 1520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:36.475504 1522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:36.475759 1519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:36.475968 1514 server_base.cc:1060] running on GCE node
I20260203 21:06:36.476938 1514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:36.478158 1514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:36.479471 1514 hybrid_clock.cc:648] HybridClock initialized: now 1770152796479448 us; error 41 us; skew 500 ppm
Feb 03 21:06:36 dist-test-slave-j889 krb5kdc[1498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:36.482579 1514 init.cc:377] Logged in from keytab as kudu/127.0.102.190@KRBTEST.COM (short username kudu)
I20260203 21:06:36.483950 1514 webserver.cc:492] Webserver started at http://127.0.102.190:46285/ using document root <none> and password file <none>
I20260203 21:06:36.484545 1514 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:36.484604 1514 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:36.484833 1514 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:36.486675 1514 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "de4ddeb8e2fe47f2b27c416643b10684"
format_stamp: "Formatted at 2026-02-03 21:06:36 on dist-test-slave-j889"
server_key: "18bbf1b5aa47b8304726a2e14170403d"
server_key_iv: "db132644cfbb1859f3078f792da79f85"
server_key_version: "encryptionkey@0"
I20260203 21:06:36.487386 1514 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "de4ddeb8e2fe47f2b27c416643b10684"
format_stamp: "Formatted at 2026-02-03 21:06:36 on dist-test-slave-j889"
server_key: "18bbf1b5aa47b8304726a2e14170403d"
server_key_iv: "db132644cfbb1859f3078f792da79f85"
server_key_version: "encryptionkey@0"
I20260203 21:06:36.491677 1514 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260203 21:06:36.495034 1529 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:36.496439 1514 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260203 21:06:36.496593 1514 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "de4ddeb8e2fe47f2b27c416643b10684"
format_stamp: "Formatted at 2026-02-03 21:06:36 on dist-test-slave-j889"
server_key: "18bbf1b5aa47b8304726a2e14170403d"
server_key_iv: "db132644cfbb1859f3078f792da79f85"
server_key_version: "encryptionkey@0"
I20260203 21:06:36.496754 1514 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:36.524744 1514 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:36.528249 1514 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:36.528442 1514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:36.538450 1514 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:37333
I20260203 21:06:36.538712 1591 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:37333 every 8 connection(s)
I20260203 21:06:36.539575 1514 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:06:36.541572 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 1514
I20260203 21:06:36.541687 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:06:36.541931 410 external_mini_cluster.cc:1468] Setting key 3291db9f806d921a6d0c88cb6b5a6a17
I20260203 21:06:36.543390 1592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:36.549849 1592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684: Bootstrap starting.
Feb 03 21:06:36 dist-test-slave-j889 krb5kdc[1498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:36.552579 1592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:36.553828 1592 log.cc:826] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:36.556136 1592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684: No bootstrap required, opened a new log
I20260203 21:06:36.557643 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:36.543350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.0.102.190:37333)
0203 21:06:36.544022 (+ 672us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:36.544033 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:36.544071 (+ 38us) server_negotiation.cc:408] Connection header received
0203 21:06:36.544806 (+ 735us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:36.544828 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:36.545148 (+ 320us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:36.545451 (+ 303us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:36.546653 (+ 1202us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:36.547596 (+ 943us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:36.548560 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:36.548909 (+ 349us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:36.551752 (+ 2843us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:36.551807 (+ 55us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:36.551819 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:36.551869 (+ 50us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:36.553994 (+ 2125us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:36.554591 (+ 597us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:36.554598 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:36.554604 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:36.554825 (+ 221us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:36.555129 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:36.555132 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:36.555133 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:36.555526 (+ 393us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:36.555926 (+ 400us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:36.556304 (+ 378us) server_negotiation.cc:300] Negotiation successful
0203 21:06:36.556612 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":163,"threads_started":1}
I20260203 21:06:36.559510 1592 raft_consensus.cc:359] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4ddeb8e2fe47f2b27c416643b10684" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 37333 } }
I20260203 21:06:36.559748 1592 raft_consensus.cc:385] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:36.559814 1592 raft_consensus.cc:740] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4ddeb8e2fe47f2b27c416643b10684, State: Initialized, Role: FOLLOWER
I20260203 21:06:36.560334 1592 consensus_queue.cc:260] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4ddeb8e2fe47f2b27c416643b10684" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 37333 } }
I20260203 21:06:36.560482 1592 raft_consensus.cc:399] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:06:36.560544 1592 raft_consensus.cc:493] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:06:36.560667 1592 raft_consensus.cc:3060] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:36.561800 1592 raft_consensus.cc:515] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4ddeb8e2fe47f2b27c416643b10684" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 37333 } }
I20260203 21:06:36.562186 1592 leader_election.cc:304] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de4ddeb8e2fe47f2b27c416643b10684; no voters:
I20260203 21:06:36.562521 1592 leader_election.cc:290] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:06:36.562682 1597 raft_consensus.cc:2804] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:36.562914 1597 raft_consensus.cc:697] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [term 1 LEADER]: Becoming Leader. State: Replica: de4ddeb8e2fe47f2b27c416643b10684, State: Running, Role: LEADER
I20260203 21:06:36.563225 1597 consensus_queue.cc:237] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4ddeb8e2fe47f2b27c416643b10684" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 37333 } }
I20260203 21:06:36.563800 1592 sys_catalog.cc:565] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:06:36.565003 1598 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de4ddeb8e2fe47f2b27c416643b10684" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4ddeb8e2fe47f2b27c416643b10684" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 37333 } } }
I20260203 21:06:36.565141 1598 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [sys.catalog]: This master's current role is: LEADER
I20260203 21:06:36.565399 1599 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de4ddeb8e2fe47f2b27c416643b10684. Latest consensus state: current_term: 1 leader_uuid: "de4ddeb8e2fe47f2b27c416643b10684" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4ddeb8e2fe47f2b27c416643b10684" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 37333 } } }
I20260203 21:06:36.565486 1599 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684 [sys.catalog]: This master's current role is: LEADER
I20260203 21:06:36.565928 1606 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:06:36.572436 1606 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:06:36.579289 1606 catalog_manager.cc:1357] Generated new cluster ID: dc553da0ea234183b427c6d0ac0972b0
I20260203 21:06:36.579380 1606 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:06:36.593116 1606 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:06:36.594489 1606 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:06:36.609174 1606 catalog_manager.cc:6027] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684: Generated new TSK 0
I20260203 21:06:36.610128 1606 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:36.685742 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:37333
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:46633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:36.814824 1620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:36.815086 1620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:36.815182 1620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:36.819502 1620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:36.819624 1620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:36.819706 1620 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260203 21:06:36.819869 1620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:06:36.824884 1620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.102.190:37333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:36.826087 1620 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:36.827004 1620 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:36.836357 1625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:36.836356 1628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:36.836546 1626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:36.836733 1620 server_base.cc:1060] running on GCE node
I20260203 21:06:36.837248 1620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:36.837855 1620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:36.839051 1620 hybrid_clock.cc:648] HybridClock initialized: now 1770152796838986 us; error 79 us; skew 500 ppm
Feb 03 21:06:36 dist-test-slave-j889 krb5kdc[1498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:36.842451 1620 init.cc:377] Logged in from keytab as kudu/127.0.102.129@KRBTEST.COM (short username kudu)
I20260203 21:06:36.843670 1620 webserver.cc:492] Webserver started at http://127.0.102.129:33031/ using document root <none> and password file <none>
I20260203 21:06:36.844260 1620 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:36.844316 1620 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:36.844509 1620 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:36.846390 1620 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "3cfa5e1ee9534cc58871e84df53170b6"
format_stamp: "Formatted at 2026-02-03 21:06:36 on dist-test-slave-j889"
server_key: "9e0b1ef1db69f97dafed56e7552fb7cf"
server_key_iv: "50bedeae37f09e013affc938f93c1810"
server_key_version: "encryptionkey@0"
I20260203 21:06:36.847290 1620 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "3cfa5e1ee9534cc58871e84df53170b6"
format_stamp: "Formatted at 2026-02-03 21:06:36 on dist-test-slave-j889"
server_key: "9e0b1ef1db69f97dafed56e7552fb7cf"
server_key_iv: "50bedeae37f09e013affc938f93c1810"
server_key_version: "encryptionkey@0"
I20260203 21:06:36.851377 1620 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260203 21:06:36.854809 1635 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:36.856516 1620 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260203 21:06:36.856691 1620 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "3cfa5e1ee9534cc58871e84df53170b6"
format_stamp: "Formatted at 2026-02-03 21:06:36 on dist-test-slave-j889"
server_key: "9e0b1ef1db69f97dafed56e7552fb7cf"
server_key_iv: "50bedeae37f09e013affc938f93c1810"
server_key_version: "encryptionkey@0"
I20260203 21:06:36.856992 1620 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:36.878815 1620 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:36.882074 1620 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:36.882262 1620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:36.883762 1620 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:36.883826 1620 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:36.883878 1620 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:36.883905 1620 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 03 21:06:36 dist-test-slave-j889 krb5kdc[1498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:36.896329 1750 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:34843 every 8 connection(s)
I20260203 21:06:36.896378 1620 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:34843
I20260203 21:06:36.897746 1620 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:06:36.899842 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:36.884512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:44007 (local address 127.0.102.190:37333)
0203 21:06:36.884654 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:36.884659 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:36.885607 (+ 948us) server_negotiation.cc:408] Connection header received
0203 21:06:36.886540 (+ 933us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:36.886544 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:36.886604 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:36.886673 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:36.888469 (+ 1796us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:36.889307 (+ 838us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:36.890415 (+ 1108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:36.890715 (+ 300us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:36.895875 (+ 5160us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:36.895908 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:36.895911 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:36.895949 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:36.897678 (+ 1729us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:36.898368 (+ 690us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:36.898372 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:36.898374 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:36.898438 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:36.898723 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:36.898726 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:36.898728 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:36.898914 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:36.899005 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:36.899518 (+ 513us) server_negotiation.cc:300] Negotiation successful
0203 21:06:36.899666 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260203 21:06:36.900436 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:36.884901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:37333 (local address 127.0.102.129:44007)
0203 21:06:36.885450 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:36.885488 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:36.886305 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:36.886873 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:36.886892 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:36.887501 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:36.888263 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:36.888284 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:36.889492 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:36.889497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:36.890150 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:36.890162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:36.890603 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:36.892541 (+ 1938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:36.892598 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:36.895607 (+ 3009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:36.897912 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:36.897918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:36.897928 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:36.898224 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:36.898541 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:36.898543 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:36.898545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:36.898632 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:36.899036 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:36.899043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:36.899281 (+ 238us) client_negotiation.cc:769] Sending connection context
0203 21:06:36.899468 (+ 187us) client_negotiation.cc:240] Negotiation successful
0203 21:06:36.899750 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":169,"threads_started":1}
I20260203 21:06:36.901867 1751 heartbeater.cc:344] Connected to a master server at 127.0.102.190:37333
I20260203 21:06:36.902042 1751 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:36.902629 1751 heartbeater.cc:507] Master 127.0.102.190:37333 requested a full tablet report, sending...
I20260203 21:06:36.903544 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 1620
I20260203 21:06:36.903652 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:06:36.903925 410 external_mini_cluster.cc:1468] Setting key b42134dbf143d35785c77ccd7f059de5
I20260203 21:06:36.904282 1546 ts_manager.cc:194] Registered new tserver with Master: 3cfa5e1ee9534cc58871e84df53170b6 (127.0.102.129:34843)
I20260203 21:06:36.905480 1546 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:44007
I20260203 21:06:36.911166 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:36.903975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:36775 (local address 127.0.102.190:37333)
0203 21:06:36.904094 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:36.904099 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:36.904539 (+ 440us) server_negotiation.cc:408] Connection header received
0203 21:06:36.904693 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:36.904697 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:36.904867 (+ 170us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:36.904964 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:36.905553 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:36.906141 (+ 588us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:36.906905 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:36.907100 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:36.908298 (+ 1198us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:36.908314 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:36.908316 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:36.908340 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:36.909771 (+ 1431us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:36.910166 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:36.910169 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:36.910171 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:36.910215 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:36.910417 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:36.910421 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:36.910423 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:36.910573 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:36.910657 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:36.910902 (+ 245us) server_negotiation.cc:300] Negotiation successful
0203 21:06:36.911010 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260203 21:06:36.911182 1757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:36.903919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:37333 (local address 127.0.102.129:36775)
0203 21:06:36.904465 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:36.904478 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:36.904574 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:36.905003 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:36.905006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:36.905219 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:36.905436 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:36.905443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:36.906243 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:36.906246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:36.906770 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:36.906781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:36.907221 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:36.907770 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:36.907787 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:36.908172 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:36.909889 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:36.909893 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:36.909895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:36.910076 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:36.910293 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:36.910297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:36.910298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:36.910344 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:36.910658 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:36.910661 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:36.910748 (+ 87us) client_negotiation.cc:769] Sending connection context
0203 21:06:36.910841 (+ 93us) client_negotiation.cc:240] Negotiation successful
0203 21:06:36.910987 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":227,"threads_started":1}
WARNING: no policy specified for kudu/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:36.957266 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:37333
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:46633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:37.070837 1762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:37.071106 1762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:37.071170 1762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:37.074826 1762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:37.074908 1762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:37.074965 1762 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260203 21:06:37.075018 1762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:06:37.079771 1762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.102.190:37333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:37.081050 1762 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:37.081960 1762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:37.090145 1770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:37.090096 1767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:37.090080 1768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:37.090404 1762 server_base.cc:1060] running on GCE node
I20260203 21:06:37.090889 1762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:37.091497 1762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:37.092708 1762 hybrid_clock.cc:648] HybridClock initialized: now 1770152797092689 us; error 35 us; skew 500 ppm
Feb 03 21:06:37 dist-test-slave-j889 krb5kdc[1498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:37.095719 1762 init.cc:377] Logged in from keytab as kudu/127.0.102.130@KRBTEST.COM (short username kudu)
I20260203 21:06:37.096961 1762 webserver.cc:492] Webserver started at http://127.0.102.130:33803/ using document root <none> and password file <none>
I20260203 21:06:37.097527 1762 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:37.097579 1762 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:37.097761 1762 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:37.099542 1762 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "3a05cb7f09c345918db000d4ca32dd08"
format_stamp: "Formatted at 2026-02-03 21:06:37 on dist-test-slave-j889"
server_key: "10ea39a190f2d5919ed3839f4da118c8"
server_key_iv: "03ef0780f049f74f997aeaead6d63582"
server_key_version: "encryptionkey@0"
I20260203 21:06:37.100039 1762 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "3a05cb7f09c345918db000d4ca32dd08"
format_stamp: "Formatted at 2026-02-03 21:06:37 on dist-test-slave-j889"
server_key: "10ea39a190f2d5919ed3839f4da118c8"
server_key_iv: "03ef0780f049f74f997aeaead6d63582"
server_key_version: "encryptionkey@0"
I20260203 21:06:37.103865 1762 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260203 21:06:37.106494 1777 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:37.107743 1762 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:06:37.107913 1762 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "3a05cb7f09c345918db000d4ca32dd08"
format_stamp: "Formatted at 2026-02-03 21:06:37 on dist-test-slave-j889"
server_key: "10ea39a190f2d5919ed3839f4da118c8"
server_key_iv: "03ef0780f049f74f997aeaead6d63582"
server_key_version: "encryptionkey@0"
I20260203 21:06:37.108027 1762 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:37.120270 1762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:37.123565 1762 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:37.123780 1762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:37.125455 1762 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:37.125517 1762 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:37.125572 1762 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:37.125598 1762 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 03 21:06:37 dist-test-slave-j889 krb5kdc[1498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:37.136679 1762 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:44321
I20260203 21:06:37.136708 1892 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:44321 every 8 connection(s)
I20260203 21:06:37.137929 1762 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:06:37.138638 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.126185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:56293 (local address 127.0.102.190:37333)
0203 21:06:37.126310 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:37.126314 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:37.127267 (+ 953us) server_negotiation.cc:408] Connection header received
0203 21:06:37.128226 (+ 959us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:37.128230 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:37.128290 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:37.128368 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:37.130106 (+ 1738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.130693 (+ 587us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.131426 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.131599 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.134486 (+ 2887us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:37.134513 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:37.134516 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:37.134552 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:37.136296 (+ 1744us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:37.136976 (+ 680us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:37.136980 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:37.136983 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:37.137047 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:37.137401 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:37.137404 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:37.137406 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:37.137616 (+ 210us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:37.137722 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:37.138317 (+ 595us) server_negotiation.cc:300] Negotiation successful
0203 21:06:37.138471 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260203 21:06:37.139271 1786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.126503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:37333 (local address 127.0.102.130:56293)
0203 21:06:37.127052 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:37.127092 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:37.127987 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:37.128679 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:37.128691 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:37.129256 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:37.129927 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.129944 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.130815 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.130819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:37.131304 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.131313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.131541 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.132219 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:37.132262 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:37.134292 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:37.136464 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:37.136470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:37.136483 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:37.136836 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:37.137181 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:37.137184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:37.137186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:37.137292 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:37.137738 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:37.137745 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:37.138032 (+ 287us) client_negotiation.cc:769] Sending connection context
0203 21:06:37.138224 (+ 192us) client_negotiation.cc:240] Negotiation successful
0203 21:06:37.138518 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":146,"threads_started":1}
I20260203 21:06:37.140502 1893 heartbeater.cc:344] Connected to a master server at 127.0.102.190:37333
I20260203 21:06:37.140712 1893 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:37.141275 1893 heartbeater.cc:507] Master 127.0.102.190:37333 requested a full tablet report, sending...
I20260203 21:06:37.142351 1543 ts_manager.cc:194] Registered new tserver with Master: 3a05cb7f09c345918db000d4ca32dd08 (127.0.102.130:44321)
I20260203 21:06:37.143057 1543 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.130@KRBTEST.COM'} at 127.0.102.130:56293
I20260203 21:06:37.143719 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 1762
I20260203 21:06:37.143824 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:06:37.144083 410 external_mini_cluster.cc:1468] Setting key 3ac0138bbad8ffbbb4f9a9b5678b32e2
I20260203 21:06:37.149722 1899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.142653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:37333 (local address 127.0.102.130:56977)
0203 21:06:37.142958 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:37.142971 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:37.143064 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:37.143342 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:37.143346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:37.143563 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:37.143882 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.143893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.144764 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.144768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:37.145323 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.145334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.145452 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.145954 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:37.145968 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:37.146266 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:37.148186 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:37.148190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:37.148191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:37.148378 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:37.148683 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:37.148688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:37.148690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:37.148755 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:37.149183 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:37.149188 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:37.149278 (+ 90us) client_negotiation.cc:769] Sending connection context
0203 21:06:37.149370 (+ 92us) client_negotiation.cc:240] Negotiation successful
0203 21:06:37.149530 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":101,"threads_started":1}
I20260203 21:06:37.149704 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.142742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:56977 (local address 127.0.102.190:37333)
0203 21:06:37.142882 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:37.142886 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:37.143035 (+ 149us) server_negotiation.cc:408] Connection header received
0203 21:06:37.143148 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:37.143151 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:37.143193 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:37.143283 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:37.144015 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.144655 (+ 640us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.145434 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.145606 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.146391 (+ 785us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:37.146409 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:37.146412 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:37.146439 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:37.148012 (+ 1573us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:37.148474 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:37.148478 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:37.148481 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:37.148534 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:37.148877 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:37.148880 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:37.148882 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:37.149083 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:37.149181 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:37.149433 (+ 252us) server_negotiation.cc:300] Negotiation successful
0203 21:06:37.149550 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:37.200132 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:37333
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:46633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:37.312827 1904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:37.313102 1904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:37.313166 1904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:37.316663 1904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:37.316757 1904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:37.316835 1904 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260203 21:06:37.316910 1904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:06:37.321836 1904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.102.190:37333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:37.323015 1904 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:37.323913 1904 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:37.331948 1912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:37.331965 1909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:37.331970 1910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:37.332038 1904 server_base.cc:1060] running on GCE node
I20260203 21:06:37.332799 1904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:37.333418 1904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:37.334607 1904 hybrid_clock.cc:648] HybridClock initialized: now 1770152797334590 us; error 34 us; skew 500 ppm
Feb 03 21:06:37 dist-test-slave-j889 krb5kdc[1498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:37.337649 1904 init.cc:377] Logged in from keytab as kudu/127.0.102.131@KRBTEST.COM (short username kudu)
I20260203 21:06:37.338865 1904 webserver.cc:492] Webserver started at http://127.0.102.131:36091/ using document root <none> and password file <none>
I20260203 21:06:37.339457 1904 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:37.339512 1904 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:37.339697 1904 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:37.341583 1904 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "48f49653806446e9b1363c915295951c"
format_stamp: "Formatted at 2026-02-03 21:06:37 on dist-test-slave-j889"
server_key: "7bcb1a0ddff95ded20571e63bba77677"
server_key_iv: "e2c57e280bde1eb00b9ad22ce7089d45"
server_key_version: "encryptionkey@0"
I20260203 21:06:37.342149 1904 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "48f49653806446e9b1363c915295951c"
format_stamp: "Formatted at 2026-02-03 21:06:37 on dist-test-slave-j889"
server_key: "7bcb1a0ddff95ded20571e63bba77677"
server_key_iv: "e2c57e280bde1eb00b9ad22ce7089d45"
server_key_version: "encryptionkey@0"
I20260203 21:06:37.345974 1904 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260203 21:06:37.348692 1919 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:37.350006 1904 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260203 21:06:37.350171 1904 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "48f49653806446e9b1363c915295951c"
format_stamp: "Formatted at 2026-02-03 21:06:37 on dist-test-slave-j889"
server_key: "7bcb1a0ddff95ded20571e63bba77677"
server_key_iv: "e2c57e280bde1eb00b9ad22ce7089d45"
server_key_version: "encryptionkey@0"
I20260203 21:06:37.350286 1904 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:37.365669 1904 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:37.368959 1904 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:37.369154 1904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:37.370654 1904 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:37.370715 1904 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:37.370782 1904 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:37.370811 1904 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 03 21:06:37 dist-test-slave-j889 krb5kdc[1498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:37.381625 1904 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:38827
I20260203 21:06:37.382808 1904 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
I20260203 21:06:37.383296 2034 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:38827 every 8 connection(s)
I20260203 21:06:37.384308 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.371498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:47131 (local address 127.0.102.190:37333)
0203 21:06:37.371622 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:37.371625 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:37.372562 (+ 937us) server_negotiation.cc:408] Connection header received
0203 21:06:37.373635 (+ 1073us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:37.373638 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:37.373702 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:37.373777 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:37.375608 (+ 1831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.376347 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.377296 (+ 949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.377495 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.380267 (+ 2772us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:37.380297 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:37.380301 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:37.380338 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:37.382059 (+ 1721us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:37.382570 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:37.382574 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:37.382576 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:37.382631 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:37.383042 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:37.383047 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:37.383136 (+ 89us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:37.383365 (+ 229us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:37.383477 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:37.384072 (+ 595us) server_negotiation.cc:300] Negotiation successful
0203 21:06:37.384187 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260203 21:06:37.384981 1928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.371859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:37333 (local address 127.0.102.131:47131)
0203 21:06:37.372391 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:37.372432 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:37.373356 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:37.374082 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:37.374095 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:37.374664 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:37.375448 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.375467 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.376476 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.376480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:37.377143 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.377186 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.377455 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.378216 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:37.378251 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:37.380094 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:37.382192 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:37.382198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:37.382207 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:37.382458 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:37.382765 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:37.382769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:37.382770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:37.382910 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:37.383532 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:37.383539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:37.383807 (+ 268us) client_negotiation.cc:769] Sending connection context
0203 21:06:37.384018 (+ 211us) client_negotiation.cc:240] Negotiation successful
0203 21:06:37.384250 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":155,"threads_started":1}
I20260203 21:06:37.386233 2035 heartbeater.cc:344] Connected to a master server at 127.0.102.190:37333
I20260203 21:06:37.386410 2035 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:37.386458 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 1904
I20260203 21:06:37.386595 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:06:37.386816 410 external_mini_cluster.cc:1468] Setting key 51e13027f5d377c70a7d3449918d5c5d
I20260203 21:06:37.387060 2035 heartbeater.cc:507] Master 127.0.102.190:37333 requested a full tablet report, sending...
I20260203 21:06:37.388258 1543 ts_manager.cc:194] Registered new tserver with Master: 48f49653806446e9b1363c915295951c (127.0.102.131:38827)
I20260203 21:06:37.389029 1543 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.131@KRBTEST.COM'} at 127.0.102.131:47131
I20260203 21:06:37.389156 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260203 21:06:37.395704 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.388962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:36119 (local address 127.0.102.190:37333)
0203 21:06:37.389084 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:37.389087 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:37.389359 (+ 272us) server_negotiation.cc:408] Connection header received
0203 21:06:37.389517 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:37.389520 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:37.389561 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:37.389626 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:37.390195 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.390673 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.391461 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.391611 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.392524 (+ 913us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:37.392548 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:37.392552 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:37.392581 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:37.394146 (+ 1565us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:37.394630 (+ 484us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:37.394633 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:37.394635 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:37.394681 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:37.394909 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:37.394911 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:37.394912 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:37.395070 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:37.395164 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:37.395439 (+ 275us) server_negotiation.cc:300] Negotiation successful
0203 21:06:37.395540 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260203 21:06:37.395723 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.388894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:37333 (local address 127.0.102.131:36119)
0203 21:06:37.389280 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:37.389296 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:37.389424 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:37.389653 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:37.389656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:37.389844 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:37.390065 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.390072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.390781 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.390784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:37.391322 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.391335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.391463 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.392056 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:37.392072 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:37.392380 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:37.394292 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:37.394296 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:37.394298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:37.394528 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:37.394762 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:37.394765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:37.394768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:37.394823 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:37.395190 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:37.395193 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:37.395282 (+ 89us) client_negotiation.cc:769] Sending connection context
0203 21:06:37.395377 (+ 95us) client_negotiation.cc:240] Negotiation successful
0203 21:06:37.395535 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":119,"threads_started":1}
Feb 03 21:06:37 dist-test-slave-j889 krb5kdc[1498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 03 21:06:37 dist-test-slave-j889 krb5kdc[1498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:37.414213 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.406677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.0.102.190:37333)
0203 21:06:37.406868 (+ 191us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:37.406872 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:37.406885 (+ 13us) server_negotiation.cc:408] Connection header received
0203 21:06:37.406924 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:37.406927 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:37.406971 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:37.407047 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:37.407873 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.408372 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.409071 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.409250 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.411343 (+ 2093us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:37.411368 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:37.411370 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:37.411404 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:37.412768 (+ 1364us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:37.413176 (+ 408us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:37.413179 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:37.413181 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:37.413230 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:37.413432 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:37.413436 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:37.413438 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:37.413598 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:37.413717 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:37.413935 (+ 218us) server_negotiation.cc:300] Negotiation successful
0203 21:06:37.414046 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260203 21:06:37.417220 1543 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260203 21:06:37.419499 1543 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:06:37.431391 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.427472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:38827 (local address 127.0.0.1:54726)
0203 21:06:37.427789 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:37.427829 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:37.427972 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:37.428660 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:37.428667 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:37.428689 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:37.428977 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.428985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.430248 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.430252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:37.430911 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.430921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.431011 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.431067 (+ 56us) client_negotiation.cc:769] Sending connection context
0203 21:06:37.431158 (+ 91us) client_negotiation.cc:240] Negotiation successful
0203 21:06:37.431246 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":83,"threads_started":1}
I20260203 21:06:37.432431 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.427630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54726 (local address 127.0.102.131:38827)
0203 21:06:37.427966 (+ 336us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:37.427973 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:37.427997 (+ 24us) server_negotiation.cc:408] Connection header received
0203 21:06:37.428111 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:37.428117 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:37.428299 (+ 182us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:37.428441 (+ 142us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:37.429122 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.430154 (+ 1032us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.431535 (+ 1381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.432051 (+ 516us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.432157 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:37.432210 (+ 53us) server_negotiation.cc:300] Negotiation successful
0203 21:06:37.432293 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":111,"threads_started":1}
I20260203 21:06:37.432521 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.428173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:44321 (local address 127.0.0.1:39144)
0203 21:06:37.428703 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:37.428716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:37.428787 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:37.429435 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:37.429438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:37.429456 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:37.429715 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.429721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.431203 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.431207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:37.432140 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.432151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.432265 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.432282 (+ 17us) client_negotiation.cc:769] Sending connection context
0203 21:06:37.432328 (+ 46us) client_negotiation.cc:240] Negotiation successful
0203 21:06:37.432386 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":89,"threads_started":1}
I20260203 21:06:37.433280 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.428339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.0.102.130:44321)
0203 21:06:37.429039 (+ 700us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:37.429046 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:37.429067 (+ 21us) server_negotiation.cc:408] Connection header received
0203 21:06:37.429145 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:37.429150 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:37.429311 (+ 161us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:37.429438 (+ 127us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:37.429830 (+ 392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.431043 (+ 1213us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.432289 (+ 1246us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.432970 (+ 681us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.433058 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:37.433112 (+ 54us) server_negotiation.cc:300] Negotiation successful
0203 21:06:37.433172 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":91,"threads_started":1}
I20260203 21:06:37.432797 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.428481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:34843 (local address 127.0.0.1:60710)
0203 21:06:37.428941 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:37.428954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:37.429038 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:37.429599 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:37.429602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:37.429619 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:37.429875 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.429881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.431270 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.431273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:37.432181 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.432188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.432265 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.432282 (+ 17us) client_negotiation.cc:769] Sending connection context
0203 21:06:37.432328 (+ 46us) client_negotiation.cc:240] Negotiation successful
0203 21:06:37.432386 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":80,"threads_started":1}
I20260203 21:06:37.433444 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.428865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.0.102.129:34843)
0203 21:06:37.429201 (+ 336us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:37.429205 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:37.429222 (+ 17us) server_negotiation.cc:408] Connection header received
0203 21:06:37.429296 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:37.429300 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:37.429438 (+ 138us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:37.429565 (+ 127us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:37.429994 (+ 429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.431164 (+ 1170us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.432472 (+ 1308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.433067 (+ 595us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.433171 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:37.433232 (+ 61us) server_negotiation.cc:300] Negotiation successful
0203 21:06:37.433315 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":74,"threads_started":1}
I20260203 21:06:37.435504 1969 tablet_service.cc:1505] Processing CreateTablet for tablet dac92a2930ae4dfda72e17e74dec839a (DEFAULT_TABLE table=test-table [id=fa2fc89e2e634d9380d71f246488e91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:37.436457 1969 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dac92a2930ae4dfda72e17e74dec839a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:37.436360 1683 tablet_service.cc:1505] Processing CreateTablet for tablet dac92a2930ae4dfda72e17e74dec839a (DEFAULT_TABLE table=test-table [id=fa2fc89e2e634d9380d71f246488e91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:37.436360 1827 tablet_service.cc:1505] Processing CreateTablet for tablet dac92a2930ae4dfda72e17e74dec839a (DEFAULT_TABLE table=test-table [id=fa2fc89e2e634d9380d71f246488e91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:37.437379 1683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dac92a2930ae4dfda72e17e74dec839a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:37.437379 1827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dac92a2930ae4dfda72e17e74dec839a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:37.442276 2058 tablet_bootstrap.cc:492] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c: Bootstrap starting.
I20260203 21:06:37.443892 2060 tablet_bootstrap.cc:492] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08: Bootstrap starting.
I20260203 21:06:37.444209 2059 tablet_bootstrap.cc:492] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6: Bootstrap starting.
I20260203 21:06:37.445375 2058 tablet_bootstrap.cc:654] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:37.445914 2059 tablet_bootstrap.cc:654] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:37.446065 2060 tablet_bootstrap.cc:654] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:37.446568 2059 log.cc:826] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:37.446640 2058 log.cc:826] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:37.447163 2060 log.cc:826] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:37.448319 2059 tablet_bootstrap.cc:492] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6: No bootstrap required, opened a new log
I20260203 21:06:37.448346 2058 tablet_bootstrap.cc:492] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c: No bootstrap required, opened a new log
I20260203 21:06:37.448504 2059 ts_tablet_manager.cc:1403] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260203 21:06:37.448505 2058 ts_tablet_manager.cc:1403] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260203 21:06:37.449193 2060 tablet_bootstrap.cc:492] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08: No bootstrap required, opened a new log
I20260203 21:06:37.449514 2060 ts_tablet_manager.cc:1403] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260203 21:06:37.451025 2059 raft_consensus.cc:359] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } }
I20260203 21:06:37.451222 2059 raft_consensus.cc:385] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:37.451292 2059 raft_consensus.cc:740] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cfa5e1ee9534cc58871e84df53170b6, State: Initialized, Role: FOLLOWER
I20260203 21:06:37.451547 2058 raft_consensus.cc:359] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } }
I20260203 21:06:37.451764 2058 raft_consensus.cc:385] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:37.451723 2059 consensus_queue.cc:260] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } }
I20260203 21:06:37.451880 2058 raft_consensus.cc:740] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f49653806446e9b1363c915295951c, State: Initialized, Role: FOLLOWER
I20260203 21:06:37.452296 2058 consensus_queue.cc:260] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } }
I20260203 21:06:37.452502 2059 ts_tablet_manager.cc:1434] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260203 21:06:37.452430 2060 raft_consensus.cc:359] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } }
I20260203 21:06:37.452672 2060 raft_consensus.cc:385] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:37.452734 2060 raft_consensus.cc:740] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a05cb7f09c345918db000d4ca32dd08, State: Initialized, Role: FOLLOWER
I20260203 21:06:37.453040 1751 heartbeater.cc:499] Master 127.0.102.190:37333 was elected leader, sending a full tablet report...
I20260203 21:06:37.453148 2058 ts_tablet_manager.cc:1434] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260203 21:06:37.453147 2060 consensus_queue.cc:260] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } }
I20260203 21:06:37.453464 2035 heartbeater.cc:499] Master 127.0.102.190:37333 was elected leader, sending a full tablet report...
I20260203 21:06:37.453816 2060 ts_tablet_manager.cc:1434] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260203 21:06:37.454101 1893 heartbeater.cc:499] Master 127.0.102.190:37333 was elected leader, sending a full tablet report...
W20260203 21:06:37.633906 2036 tablet.cc:2378] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260203 21:06:37.639321 1894 tablet.cc:2378] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260203 21:06:37.649240 1752 tablet.cc:2378] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260203 21:06:37.683637 2065 raft_consensus.cc:493] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:06:37.683831 2065 raft_consensus.cc:515] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } }
I20260203 21:06:37.684939 2065 leader_election.cc:290] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a05cb7f09c345918db000d4ca32dd08 (127.0.102.130:44321), 3cfa5e1ee9534cc58871e84df53170b6 (127.0.102.129:34843)
I20260203 21:06:37.688067 1928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.685277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:44321 (local address 127.0.102.131:54877)
0203 21:06:37.685416 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:37.685432 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:37.685549 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:37.685838 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:37.685841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:37.685861 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:37.686071 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.686077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.686826 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.686829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:37.687733 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.687742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.687849 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.687869 (+ 20us) client_negotiation.cc:769] Sending connection context
0203 21:06:37.687906 (+ 37us) client_negotiation.cc:240] Negotiation successful
0203 21:06:37.687960 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260203 21:06:37.688608 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.685371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:54877 (local address 127.0.102.130:44321)
0203 21:06:37.685539 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:37.685544 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:37.685562 (+ 18us) server_negotiation.cc:408] Connection header received
0203 21:06:37.685661 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:37.685665 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:37.685723 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:37.685801 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:37.686182 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.686688 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.687855 (+ 1167us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.688341 (+ 486us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.688381 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:37.688430 (+ 49us) server_negotiation.cc:300] Negotiation successful
0203 21:06:37.688490 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260203 21:06:37.689203 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.685544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:34843 (local address 127.0.102.131:46757)
0203 21:06:37.685879 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:37.685889 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:37.685984 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:37.686210 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:37.686213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:37.686227 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:37.686465 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.686471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.687855 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.687859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:37.688843 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.688852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.688963 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.688977 (+ 14us) client_negotiation.cc:769] Sending connection context
0203 21:06:37.689017 (+ 40us) client_negotiation.cc:240] Negotiation successful
0203 21:06:37.689073 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":130,"threads_started":1}
I20260203 21:06:37.689353 1847 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dac92a2930ae4dfda72e17e74dec839a" candidate_uuid: "48f49653806446e9b1363c915295951c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a05cb7f09c345918db000d4ca32dd08" is_pre_election: true
I20260203 21:06:37.689623 1847 raft_consensus.cc:2468] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48f49653806446e9b1363c915295951c in term 0.
I20260203 21:06:37.689656 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.685662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:46757 (local address 127.0.102.129:34843)
0203 21:06:37.685785 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:37.685789 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:37.685967 (+ 178us) server_negotiation.cc:408] Connection header received
0203 21:06:37.686090 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:37.686092 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:37.686134 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:37.686189 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:37.686568 (+ 379us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.687724 (+ 1156us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.688956 (+ 1232us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.689429 (+ 473us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.689465 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:37.689519 (+ 54us) server_negotiation.cc:300] Negotiation successful
0203 21:06:37.689572 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260203 21:06:37.690078 1921 leader_election.cc:304] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a05cb7f09c345918db000d4ca32dd08, 48f49653806446e9b1363c915295951c; no voters:
I20260203 21:06:37.690243 1705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dac92a2930ae4dfda72e17e74dec839a" candidate_uuid: "48f49653806446e9b1363c915295951c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cfa5e1ee9534cc58871e84df53170b6" is_pre_election: true
I20260203 21:06:37.690326 2065 raft_consensus.cc:2804] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:06:37.690416 2065 raft_consensus.cc:493] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:06:37.690467 2065 raft_consensus.cc:3060] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:37.690478 1705 raft_consensus.cc:2468] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48f49653806446e9b1363c915295951c in term 0.
I20260203 21:06:37.691444 2065 raft_consensus.cc:515] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } }
I20260203 21:06:37.691776 2065 leader_election.cc:290] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [CANDIDATE]: Term 1 election: Requested vote from peers 3a05cb7f09c345918db000d4ca32dd08 (127.0.102.130:44321), 3cfa5e1ee9534cc58871e84df53170b6 (127.0.102.129:34843)
I20260203 21:06:37.692157 1847 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dac92a2930ae4dfda72e17e74dec839a" candidate_uuid: "48f49653806446e9b1363c915295951c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a05cb7f09c345918db000d4ca32dd08"
I20260203 21:06:37.692202 1705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dac92a2930ae4dfda72e17e74dec839a" candidate_uuid: "48f49653806446e9b1363c915295951c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cfa5e1ee9534cc58871e84df53170b6"
I20260203 21:06:37.692292 1705 raft_consensus.cc:3060] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:37.692276 1847 raft_consensus.cc:3060] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:37.693274 1705 raft_consensus.cc:2468] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48f49653806446e9b1363c915295951c in term 1.
I20260203 21:06:37.693284 1847 raft_consensus.cc:2468] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48f49653806446e9b1363c915295951c in term 1.
I20260203 21:06:37.693558 1921 leader_election.cc:304] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a05cb7f09c345918db000d4ca32dd08, 48f49653806446e9b1363c915295951c; no voters:
I20260203 21:06:37.693730 2065 raft_consensus.cc:2804] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:37.693961 2065 raft_consensus.cc:697] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [term 1 LEADER]: Becoming Leader. State: Replica: 48f49653806446e9b1363c915295951c, State: Running, Role: LEADER
I20260203 21:06:37.694314 2065 consensus_queue.cc:237] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } }
I20260203 21:06:37.697350 1546 catalog_manager.cc:5654] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c reported cstate change: term changed from 0 to 1, leader changed from <none> to 48f49653806446e9b1363c915295951c (127.0.102.131). New cstate: current_term: 1 leader_uuid: "48f49653806446e9b1363c915295951c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } health_report { overall_health: UNKNOWN } } }
I20260203 21:06:37.729535 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.726273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.0.102.131:38827)
0203 21:06:37.726461 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:37.726465 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:37.726485 (+ 20us) server_negotiation.cc:408] Connection header received
0203 21:06:37.726558 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:37.726561 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:37.726618 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:37.726702 (+ 84us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:06:37.727116 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.727640 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.728350 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.728509 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.728562 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:06:37.729081 (+ 519us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:06:37.729180 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:37.729352 (+ 172us) server_negotiation.cc:300] Negotiation successful
0203 21:06:37.729416 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260203 21:06:37.735517 1847 raft_consensus.cc:1275] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08 [term 1 FOLLOWER]: Refusing update from remote peer 48f49653806446e9b1363c915295951c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:06:37.735596 1705 raft_consensus.cc:1275] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6 [term 1 FOLLOWER]: Refusing update from remote peer 48f49653806446e9b1363c915295951c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:06:37.736191 2068 consensus_queue.cc:1048] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:06:37.736426 2065 consensus_queue.cc:1048] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:06:37.748895 2071 mvcc.cc:204] Tried to move back new op lower bound from 7250545859516260352 to 7250545859358408704. Current Snapshot: MvccSnapshot[applied={T|T < 7250545859516260352 or (T in {7250545859516260352})}]
I20260203 21:06:37.755748 1546 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:06:37.755949 1546 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:06:37.758630 1546 catalog_manager.cc:5941] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684: Sending DeleteTablet for 3 replicas of tablet dac92a2930ae4dfda72e17e74dec839a
I20260203 21:06:37.759639 1683 tablet_service.cc:1552] Processing DeleteTablet for tablet dac92a2930ae4dfda72e17e74dec839a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:37 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:60710
I20260203 21:06:37.759630 1969 tablet_service.cc:1552] Processing DeleteTablet for tablet dac92a2930ae4dfda72e17e74dec839a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:37 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:54726
I20260203 21:06:37.759683 1827 tablet_service.cc:1552] Processing DeleteTablet for tablet dac92a2930ae4dfda72e17e74dec839a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:37 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:39144
I20260203 21:06:37.760198 2082 tablet_replica.cc:333] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c: stopping tablet replica
I20260203 21:06:37.760497 2082 raft_consensus.cc:2243] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [term 1 LEADER]: Raft consensus shutting down.
I20260203 21:06:37.760840 2080 tablet_replica.cc:333] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08: stopping tablet replica
I20260203 21:06:37.760972 2082 raft_consensus.cc:2272] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:06:37.760957 1546 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260203 21:06:37.761475 2080 raft_consensus.cc:2243] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:06:37.761504 2081 tablet_replica.cc:333] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6: stopping tablet replica
W20260203 21:06:37.761750 1546 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:06:37.761778 2081 raft_consensus.cc:2243] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:06:37.762005 2080 raft_consensus.cc:2272] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:06:37.762090 2081 raft_consensus.cc:2272] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:06:37.763547 2080 ts_tablet_manager.cc:1916] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:06:37.763777 2081 ts_tablet_manager.cc:1916] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:06:37.766489 2080 ts_tablet_manager.cc:1929] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:06:37.766508 2081 ts_tablet_manager.cc:1929] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:06:37.766593 2081 log.cc:1199] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/wal/wals/dac92a2930ae4dfda72e17e74dec839a
I20260203 21:06:37.766728 2080 log.cc:1199] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/wal/wals/dac92a2930ae4dfda72e17e74dec839a
I20260203 21:06:37.766888 2081 ts_tablet_manager.cc:1950] T dac92a2930ae4dfda72e17e74dec839a P 3cfa5e1ee9534cc58871e84df53170b6: Deleting consensus metadata
I20260203 21:06:37.767277 2080 ts_tablet_manager.cc:1950] T dac92a2930ae4dfda72e17e74dec839a P 3a05cb7f09c345918db000d4ca32dd08: Deleting consensus metadata
I20260203 21:06:37.768195 1683 tablet_service.cc:1505] Processing CreateTablet for tablet a551d7919f7741aca817ce78e04cbbfd (DEFAULT_TABLE table=test-table [id=2c24bdc7e1ed462da4ef6ad06accf26b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:37.768421 1683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a551d7919f7741aca817ce78e04cbbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:37.768939 1969 tablet_service.cc:1505] Processing CreateTablet for tablet a551d7919f7741aca817ce78e04cbbfd (DEFAULT_TABLE table=test-table [id=2c24bdc7e1ed462da4ef6ad06accf26b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:37.769204 1969 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a551d7919f7741aca817ce78e04cbbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:37.770396 2059 tablet_bootstrap.cc:492] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6: Bootstrap starting.
I20260203 21:06:37.771271 2059 tablet_bootstrap.cc:654] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:37.771401 2058 tablet_bootstrap.cc:492] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c: Bootstrap starting.
I20260203 21:06:37.771884 1531 catalog_manager.cc:4985] TS 3a05cb7f09c345918db000d4ca32dd08 (127.0.102.130:44321): tablet dac92a2930ae4dfda72e17e74dec839a (table test-table [id=fa2fc89e2e634d9380d71f246488e91e]) successfully deleted
I20260203 21:06:37.772071 1531 catalog_manager.cc:4985] TS 3cfa5e1ee9534cc58871e84df53170b6 (127.0.102.129:34843): tablet dac92a2930ae4dfda72e17e74dec839a (table test-table [id=fa2fc89e2e634d9380d71f246488e91e]) successfully deleted
I20260203 21:06:37.772194 1827 tablet_service.cc:1505] Processing CreateTablet for tablet a551d7919f7741aca817ce78e04cbbfd (DEFAULT_TABLE table=test-table [id=2c24bdc7e1ed462da4ef6ad06accf26b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:37.772397 2058 tablet_bootstrap.cc:654] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:37.772445 1827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a551d7919f7741aca817ce78e04cbbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:37.773587 2082 ts_tablet_manager.cc:1916] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:06:37.774644 2060 tablet_bootstrap.cc:492] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08: Bootstrap starting.
I20260203 21:06:37.774848 2058 tablet_bootstrap.cc:492] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c: No bootstrap required, opened a new log
I20260203 21:06:37.774942 2058 ts_tablet_manager.cc:1403] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260203 21:06:37.775457 2058 raft_consensus.cc:359] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.775528 2059 tablet_bootstrap.cc:492] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6: No bootstrap required, opened a new log
I20260203 21:06:37.775568 2058 raft_consensus.cc:385] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:37.775607 2059 ts_tablet_manager.cc:1403] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260203 21:06:37.775610 2058 raft_consensus.cc:740] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f49653806446e9b1363c915295951c, State: Initialized, Role: FOLLOWER
I20260203 21:06:37.775744 2058 consensus_queue.cc:260] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.776002 2058 ts_tablet_manager.cc:1434] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:06:37.776100 2059 raft_consensus.cc:359] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.776206 2059 raft_consensus.cc:385] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:37.776240 2059 raft_consensus.cc:740] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cfa5e1ee9534cc58871e84df53170b6, State: Initialized, Role: FOLLOWER
I20260203 21:06:37.776223 2060 tablet_bootstrap.cc:654] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:37.776361 2059 consensus_queue.cc:260] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.776609 2059 ts_tablet_manager.cc:1434] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:06:37.776768 2082 ts_tablet_manager.cc:1929] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:06:37.776885 2082 log.cc:1199] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/wal/wals/dac92a2930ae4dfda72e17e74dec839a
I20260203 21:06:37.777222 2082 ts_tablet_manager.cc:1950] T dac92a2930ae4dfda72e17e74dec839a P 48f49653806446e9b1363c915295951c: Deleting consensus metadata
I20260203 21:06:37.778329 1531 catalog_manager.cc:4985] TS 48f49653806446e9b1363c915295951c (127.0.102.131:38827): tablet dac92a2930ae4dfda72e17e74dec839a (table test-table [id=fa2fc89e2e634d9380d71f246488e91e]) successfully deleted
I20260203 21:06:37.781078 2064 raft_consensus.cc:493] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:06:37.781232 2060 tablet_bootstrap.cc:492] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08: No bootstrap required, opened a new log
I20260203 21:06:37.781181 2064 raft_consensus.cc:515] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.781330 2060 ts_tablet_manager.cc:1403] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260203 21:06:37.781806 2060 raft_consensus.cc:359] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.781910 2060 raft_consensus.cc:385] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:37.781939 2060 raft_consensus.cc:740] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a05cb7f09c345918db000d4ca32dd08, State: Initialized, Role: FOLLOWER
I20260203 21:06:37.782055 2060 consensus_queue.cc:260] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.782310 2060 ts_tablet_manager.cc:1434] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:06:37.782466 2064 leader_election.cc:290] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a05cb7f09c345918db000d4ca32dd08 (127.0.102.130:44321), 48f49653806446e9b1363c915295951c (127.0.102.131:38827)
I20260203 21:06:37.787509 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.783436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:60041 (local address 127.0.102.130:44321)
0203 21:06:37.783549 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:37.783553 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:37.783566 (+ 13us) server_negotiation.cc:408] Connection header received
0203 21:06:37.783614 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:37.783618 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:37.783675 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:37.783743 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:37.784364 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.785021 (+ 657us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.786679 (+ 1658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.787132 (+ 453us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.787167 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:37.787352 (+ 185us) server_negotiation.cc:300] Negotiation successful
0203 21:06:37.787416 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260203 21:06:37.787528 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.782621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:44321 (local address 127.0.102.129:60041)
0203 21:06:37.783236 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:37.783253 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:37.783351 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:37.783872 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:37.783876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:37.783897 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:37.784150 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.784158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.785137 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.785141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:37.786545 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.786557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.787224 (+ 667us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.787238 (+ 14us) client_negotiation.cc:769] Sending connection context
0203 21:06:37.787309 (+ 71us) client_negotiation.cc:240] Negotiation successful
0203 21:06:37.787375 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":104,"threads_started":1}
I20260203 21:06:37.788162 1847 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a551d7919f7741aca817ce78e04cbbfd" candidate_uuid: "3cfa5e1ee9534cc58871e84df53170b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a05cb7f09c345918db000d4ca32dd08" is_pre_election: true
I20260203 21:06:37.788313 1847 raft_consensus.cc:2468] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cfa5e1ee9534cc58871e84df53170b6 in term 0.
I20260203 21:06:37.788671 1637 leader_election.cc:304] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a05cb7f09c345918db000d4ca32dd08, 3cfa5e1ee9534cc58871e84df53170b6; no voters:
I20260203 21:06:37.788831 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.783074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:50183 (local address 127.0.102.131:38827)
0203 21:06:37.783356 (+ 282us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:37.783361 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:37.784992 (+ 1631us) server_negotiation.cc:408] Connection header received
0203 21:06:37.785184 (+ 192us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:37.785188 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:37.785241 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:37.785320 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:37.785773 (+ 453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.786592 (+ 819us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.787837 (+ 1245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.788359 (+ 522us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.788396 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:37.788611 (+ 215us) server_negotiation.cc:300] Negotiation successful
0203 21:06:37.788727 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260203 21:06:37.788849 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.782990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:38827 (local address 127.0.102.129:50183)
0203 21:06:37.784800 (+ 1810us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:37.784814 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:37.785064 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:37.785333 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:37.785336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:37.785354 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:37.785654 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.785663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.786735 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.786739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:37.787701 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.787711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.788458 (+ 747us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.788473 (+ 15us) client_negotiation.cc:769] Sending connection context
0203 21:06:37.788574 (+ 101us) client_negotiation.cc:240] Negotiation successful
0203 21:06:37.788660 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1733,"thread_start_us":109,"threads_started":1}
I20260203 21:06:37.788942 2064 raft_consensus.cc:2804] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:06:37.788996 2064 raft_consensus.cc:493] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:06:37.789036 2064 raft_consensus.cc:3060] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:37.789963 2064 raft_consensus.cc:515] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.790345 2064 leader_election.cc:290] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [CANDIDATE]: Term 1 election: Requested vote from peers 3a05cb7f09c345918db000d4ca32dd08 (127.0.102.130:44321), 48f49653806446e9b1363c915295951c (127.0.102.131:38827)
I20260203 21:06:37.790647 1989 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a551d7919f7741aca817ce78e04cbbfd" candidate_uuid: "3cfa5e1ee9534cc58871e84df53170b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f49653806446e9b1363c915295951c" is_pre_election: true
I20260203 21:06:37.790729 1988 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a551d7919f7741aca817ce78e04cbbfd" candidate_uuid: "3cfa5e1ee9534cc58871e84df53170b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f49653806446e9b1363c915295951c"
I20260203 21:06:37.790814 1988 raft_consensus.cc:3060] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:37.791683 1988 raft_consensus.cc:2468] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cfa5e1ee9534cc58871e84df53170b6 in term 1.
I20260203 21:06:37.792158 1847 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a551d7919f7741aca817ce78e04cbbfd" candidate_uuid: "3cfa5e1ee9534cc58871e84df53170b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a05cb7f09c345918db000d4ca32dd08"
I20260203 21:06:37.792232 1637 leader_election.cc:304] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cfa5e1ee9534cc58871e84df53170b6, 48f49653806446e9b1363c915295951c; no voters:
I20260203 21:06:37.792294 1847 raft_consensus.cc:3060] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:37.792414 2064 raft_consensus.cc:2804] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:37.792670 2064 raft_consensus.cc:697] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [term 1 LEADER]: Becoming Leader. State: Replica: 3cfa5e1ee9534cc58871e84df53170b6, State: Running, Role: LEADER
I20260203 21:06:37.792917 2064 consensus_queue.cc:237] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.793195 1847 raft_consensus.cc:2468] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cfa5e1ee9534cc58871e84df53170b6 in term 1.
I20260203 21:06:37.795276 1546 catalog_manager.cc:5654] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cfa5e1ee9534cc58871e84df53170b6 (127.0.102.129). New cstate: current_term: 1 leader_uuid: "3cfa5e1ee9534cc58871e84df53170b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } health_report { overall_health: UNKNOWN } } }
Feb 03 21:06:37 dist-test-slave-j889 krb5kdc[1498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:37.831295 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.822763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.0.102.190:37333)
0203 21:06:37.822897 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:37.822915 (+ 18us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:37.823105 (+ 190us) server_negotiation.cc:408] Connection header received
0203 21:06:37.823228 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:37.823232 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:37.823284 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:37.823376 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:37.824226 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.825051 (+ 825us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.825781 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.825985 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.828046 (+ 2061us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:37.828064 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:37.828066 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:37.828088 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:37.829588 (+ 1500us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:37.830135 (+ 547us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:37.830139 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:37.830140 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:37.830187 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:37.830442 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:37.830445 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:37.830446 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:37.830596 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:37.830675 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:37.831028 (+ 353us) server_negotiation.cc:300] Negotiation successful
0203 21:06:37.831138 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":44}
I20260203 21:06:37.831331 2095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.822697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:37333 (local address 127.0.0.1:58810)
0203 21:06:37.823024 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:37.823037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:37.823147 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:37.823420 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:37.823423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:37.823909 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:37.824108 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.824116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.825187 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.825191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:37.825648 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.825657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.825755 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.826378 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:37.826391 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:37.827910 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:37.829699 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:37.829708 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:37.829715 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:37.830018 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:37.830274 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:37.830278 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:37.830280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:37.830339 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:37.830690 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:37.830697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:37.830819 (+ 122us) client_negotiation.cc:769] Sending connection context
0203 21:06:37.830937 (+ 118us) client_negotiation.cc:240] Negotiation successful
0203 21:06:37.831082 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":125,"threads_started":1}
I20260203 21:06:37.835260 1546 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:58810:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260203 21:06:37.836229 1546 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:06:37.841763 1683 tablet_service.cc:1505] Processing CreateTablet for tablet 77a9f82cc33d4e30a2f701fc4d172368 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=706327bf0ec6427b8a7ed36d215ed6f4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260203 21:06:37.841763 1827 tablet_service.cc:1505] Processing CreateTablet for tablet 77a9f82cc33d4e30a2f701fc4d172368 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=706327bf0ec6427b8a7ed36d215ed6f4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260203 21:06:37.842176 1683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77a9f82cc33d4e30a2f701fc4d172368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:37.842176 1827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77a9f82cc33d4e30a2f701fc4d172368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:37.842062 1969 tablet_service.cc:1505] Processing CreateTablet for tablet 77a9f82cc33d4e30a2f701fc4d172368 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=706327bf0ec6427b8a7ed36d215ed6f4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260203 21:06:37.842417 1969 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77a9f82cc33d4e30a2f701fc4d172368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:37.845088 2059 tablet_bootstrap.cc:492] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6: Bootstrap starting.
I20260203 21:06:37.845229 2060 tablet_bootstrap.cc:492] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08: Bootstrap starting.
I20260203 21:06:37.846480 2060 tablet_bootstrap.cc:654] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:37.846735 2059 tablet_bootstrap.cc:654] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:37.847362 2058 tablet_bootstrap.cc:492] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c: Bootstrap starting.
I20260203 21:06:37.848053 2060 tablet_bootstrap.cc:492] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08: No bootstrap required, opened a new log
I20260203 21:06:37.848150 2060 ts_tablet_manager.cc:1403] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260203 21:06:37.848274 2058 tablet_bootstrap.cc:654] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:37.848697 2060 raft_consensus.cc:359] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.848819 2060 raft_consensus.cc:385] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:37.848850 2060 raft_consensus.cc:740] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a05cb7f09c345918db000d4ca32dd08, State: Initialized, Role: FOLLOWER
I20260203 21:06:37.848986 2060 consensus_queue.cc:260] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.849251 2060 ts_tablet_manager.cc:1434] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260203 21:06:37.849529 2059 tablet_bootstrap.cc:492] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6: No bootstrap required, opened a new log
I20260203 21:06:37.849498 2066 tablet_replica.cc:442] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } }
I20260203 21:06:37.849606 2066 tablet_replica.cc:445] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:06:37.849625 2059 ts_tablet_manager.cc:1403] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260203 21:06:37.850127 2058 tablet_bootstrap.cc:492] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c: No bootstrap required, opened a new log
I20260203 21:06:37.850142 2059 raft_consensus.cc:359] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.850215 2058 ts_tablet_manager.cc:1403] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260203 21:06:37.850252 2059 raft_consensus.cc:385] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:37.850281 2059 raft_consensus.cc:740] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cfa5e1ee9534cc58871e84df53170b6, State: Initialized, Role: FOLLOWER
I20260203 21:06:37.850405 2059 consensus_queue.cc:260] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.850668 2059 ts_tablet_manager.cc:1434] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260203 21:06:37.850659 2058 raft_consensus.cc:359] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.850754 2058 raft_consensus.cc:385] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:37.850708 2064 tablet_replica.cc:442] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } }
I20260203 21:06:37.850785 2058 raft_consensus.cc:740] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f49653806446e9b1363c915295951c, State: Initialized, Role: FOLLOWER
I20260203 21:06:37.850816 2064 tablet_replica.cc:445] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:06:37.850952 2058 consensus_queue.cc:260] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.851172 2068 tablet_replica.cc:442] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } }
I20260203 21:06:37.851296 2068 tablet_replica.cc:445] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:06:37.851183 2058 ts_tablet_manager.cc:1434] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260203 21:06:37.963477 2068 raft_consensus.cc:493] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:06:37.963632 2068 raft_consensus.cc:515] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.964063 2068 leader_election.cc:290] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cfa5e1ee9534cc58871e84df53170b6 (127.0.102.129:34843), 3a05cb7f09c345918db000d4ca32dd08 (127.0.102.130:44321)
I20260203 21:06:37.964561 1705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "77a9f82cc33d4e30a2f701fc4d172368" candidate_uuid: "48f49653806446e9b1363c915295951c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cfa5e1ee9534cc58871e84df53170b6" is_pre_election: true
I20260203 21:06:37.964604 1847 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "77a9f82cc33d4e30a2f701fc4d172368" candidate_uuid: "48f49653806446e9b1363c915295951c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a05cb7f09c345918db000d4ca32dd08" is_pre_election: true
I20260203 21:06:37.964722 1705 raft_consensus.cc:2468] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48f49653806446e9b1363c915295951c in term 0.
I20260203 21:06:37.964788 1847 raft_consensus.cc:2468] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48f49653806446e9b1363c915295951c in term 0.
I20260203 21:06:37.965016 1921 leader_election.cc:304] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cfa5e1ee9534cc58871e84df53170b6, 48f49653806446e9b1363c915295951c; no voters:
I20260203 21:06:37.965184 2068 raft_consensus.cc:2804] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:06:37.965242 2068 raft_consensus.cc:493] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:06:37.965288 2068 raft_consensus.cc:3060] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:37.966229 2068 raft_consensus.cc:515] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.966540 2068 leader_election.cc:290] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [CANDIDATE]: Term 1 election: Requested vote from peers 3cfa5e1ee9534cc58871e84df53170b6 (127.0.102.129:34843), 3a05cb7f09c345918db000d4ca32dd08 (127.0.102.130:44321)
I20260203 21:06:37.966964 1847 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "77a9f82cc33d4e30a2f701fc4d172368" candidate_uuid: "48f49653806446e9b1363c915295951c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a05cb7f09c345918db000d4ca32dd08"
I20260203 21:06:37.966955 1705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "77a9f82cc33d4e30a2f701fc4d172368" candidate_uuid: "48f49653806446e9b1363c915295951c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cfa5e1ee9534cc58871e84df53170b6"
I20260203 21:06:37.967065 1847 raft_consensus.cc:3060] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:37.967077 1705 raft_consensus.cc:3060] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:37.967876 1847 raft_consensus.cc:2468] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48f49653806446e9b1363c915295951c in term 1.
I20260203 21:06:37.967891 1705 raft_consensus.cc:2468] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48f49653806446e9b1363c915295951c in term 1.
I20260203 21:06:37.968134 1921 leader_election.cc:304] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a05cb7f09c345918db000d4ca32dd08, 48f49653806446e9b1363c915295951c; no voters:
I20260203 21:06:37.968281 2068 raft_consensus.cc:2804] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:37.968370 2068 raft_consensus.cc:697] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [term 1 LEADER]: Becoming Leader. State: Replica: 48f49653806446e9b1363c915295951c, State: Running, Role: LEADER
I20260203 21:06:37.968502 2068 consensus_queue.cc:237] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } }
I20260203 21:06:37.969154 2065 tablet_replica.cc:442] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c: TxnStatusTablet state changed. Reason: New leader 48f49653806446e9b1363c915295951c. Latest consensus state: current_term: 1 leader_uuid: "48f49653806446e9b1363c915295951c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } }
I20260203 21:06:37.969255 2065 tablet_replica.cc:445] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c: This TxnStatusTablet replica's current role is: LEADER
I20260203 21:06:37.969466 2100 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260203 21:06:37.969604 2100 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260203 21:06:37.970227 1545 catalog_manager.cc:5654] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c reported cstate change: term changed from 0 to 1, leader changed from <none> to 48f49653806446e9b1363c915295951c (127.0.102.131). New cstate: current_term: 1 leader_uuid: "48f49653806446e9b1363c915295951c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } health_report { overall_health: HEALTHY } } }
I20260203 21:06:37.984562 2095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.981570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:38827 (local address 127.0.0.1:54746)
0203 21:06:37.981725 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:37.981740 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:37.981840 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:37.982128 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:37.982131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:37.982139 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0203 21:06:37.982334 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.982340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.983314 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.983318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:37.983746 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:37.983754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.983849 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.983892 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0203 21:06:37.984289 (+ 397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0203 21:06:37.984306 (+ 17us) client_negotiation.cc:769] Sending connection context
0203 21:06:37.984376 (+ 70us) client_negotiation.cc:240] Negotiation successful
0203 21:06:37.984445 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260203 21:06:37.984584 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.981655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.0.102.131:38827)
0203 21:06:37.981812 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:37.981817 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:37.981833 (+ 16us) server_negotiation.cc:408] Connection header received
0203 21:06:37.981942 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:37.981946 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:37.982003 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:37.982084 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:06:37.982436 (+ 352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.983197 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:37.983845 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:37.984035 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:37.984080 (+ 45us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:06:37.984171 (+ 91us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:06:37.984253 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:37.984395 (+ 142us) server_negotiation.cc:300] Negotiation successful
0203 21:06:37.984450 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260203 21:06:37.986940 1847 raft_consensus.cc:1275] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08 [term 1 FOLLOWER]: Refusing update from remote peer 48f49653806446e9b1363c915295951c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:06:37.986985 1705 raft_consensus.cc:1275] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6 [term 1 FOLLOWER]: Refusing update from remote peer 48f49653806446e9b1363c915295951c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:06:37.987416 2065 consensus_queue.cc:1048] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:06:37.987560 2068 consensus_queue.cc:1048] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:06:37.989027 2064 tablet_replica.cc:442] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6: TxnStatusTablet state changed. Reason: New leader 48f49653806446e9b1363c915295951c. Latest consensus state: current_term: 1 leader_uuid: "48f49653806446e9b1363c915295951c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } }
I20260203 21:06:37.989135 2064 tablet_replica.cc:445] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:06:37.990898 2066 tablet_replica.cc:442] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08: TxnStatusTablet state changed. Reason: New leader 48f49653806446e9b1363c915295951c. Latest consensus state: current_term: 1 leader_uuid: "48f49653806446e9b1363c915295951c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } }
I20260203 21:06:37.991006 2066 tablet_replica.cc:445] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:06:37.991966 2065 tablet_replica.cc:442] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48f49653806446e9b1363c915295951c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } }
I20260203 21:06:37.992084 2065 tablet_replica.cc:445] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c: This TxnStatusTablet replica's current role is: LEADER
I20260203 21:06:37.992532 2066 tablet_replica.cc:442] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48f49653806446e9b1363c915295951c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } }
I20260203 21:06:37.992655 2066 tablet_replica.cc:445] T 77a9f82cc33d4e30a2f701fc4d172368 P 3a05cb7f09c345918db000d4ca32dd08: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:06:37.993750 2065 tablet_replica.cc:442] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48f49653806446e9b1363c915295951c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } }
I20260203 21:06:37.993850 2065 tablet_replica.cc:445] T 77a9f82cc33d4e30a2f701fc4d172368 P 48f49653806446e9b1363c915295951c: This TxnStatusTablet replica's current role is: LEADER
I20260203 21:06:37.994526 2073 mvcc.cc:204] Tried to move back new op lower bound from 7250545860550574080 to 7250545860480393216. Current Snapshot: MvccSnapshot[applied={T|T < 7250545860550574080}]
I20260203 21:06:37.994711 2064 tablet_replica.cc:442] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48f49653806446e9b1363c915295951c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cfa5e1ee9534cc58871e84df53170b6" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 34843 } } peers { permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 } } peers { permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 } } }
I20260203 21:06:37.994792 2064 tablet_replica.cc:445] T 77a9f82cc33d4e30a2f701fc4d172368 P 3cfa5e1ee9534cc58871e84df53170b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:06:38.002105 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:37.999076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.0.102.129:34843)
0203 21:06:37.999208 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:37.999212 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:37.999224 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:06:37.999299 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:37.999301 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:37.999353 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:37.999445 (+ 92us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:06:37.999778 (+ 333us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:38.000254 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:38.000916 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:38.001105 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:38.001182 (+ 77us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:06:38.001668 (+ 486us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:06:38.001780 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:38.001889 (+ 109us) server_negotiation.cc:300] Negotiation successful
0203 21:06:38.001960 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260203 21:06:38.013512 2106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:38.009942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:38827 (local address 127.0.102.129:55443)
0203 21:06:38.010251 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:38.010264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:38.010365 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:38.010624 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:38.010627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:38.010636 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0203 21:06:38.010881 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:38.010887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:38.011889 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:38.011893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:38.012593 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:38.012604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:38.012765 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:38.012813 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0203 21:06:38.013233 (+ 420us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0203 21:06:38.013239 (+ 6us) client_negotiation.cc:769] Sending connection context
0203 21:06:38.013307 (+ 68us) client_negotiation.cc:240] Negotiation successful
0203 21:06:38.013373 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":120,"threads_started":1}
I20260203 21:06:38.013518 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:38.010090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:55443 (local address 127.0.102.131:38827)
0203 21:06:38.010240 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:38.010244 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:38.010329 (+ 85us) server_negotiation.cc:408] Connection header received
0203 21:06:38.010452 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:38.010456 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:38.010514 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:38.010598 (+ 84us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:06:38.011009 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:38.011748 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:38.012767 (+ 1019us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:38.012970 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:38.013021 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:06:38.013114 (+ 93us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:06:38.013203 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:38.013331 (+ 128us) server_negotiation.cc:300] Negotiation successful
0203 21:06:38.013384 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260203 21:06:38.021579 1988 raft_consensus.cc:1275] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c [term 1 FOLLOWER]: Refusing update from remote peer 3cfa5e1ee9534cc58871e84df53170b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:06:38.021548 1847 raft_consensus.cc:1275] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08 [term 1 FOLLOWER]: Refusing update from remote peer 3cfa5e1ee9534cc58871e84df53170b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:06:38.022037 2088 consensus_queue.cc:1048] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a05cb7f09c345918db000d4ca32dd08" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 44321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:06:38.022284 2064 consensus_queue.cc:1048] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48f49653806446e9b1363c915295951c" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 38827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:06:38.026974 2072 mvcc.cc:204] Tried to move back new op lower bound from 7250545860691230720 to 7250545859762409472. Current Snapshot: MvccSnapshot[applied={T|T < 7250545860691230720}]
I20260203 21:06:38.047719 2112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:38.044162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:34843 (local address 127.0.102.131:42817)
0203 21:06:38.044618 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:38.044669 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:38.044784 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:38.045227 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:38.045230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:38.045239 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0203 21:06:38.045522 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:38.045530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:38.046391 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:38.046395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:38.046877 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:38.046886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:38.046980 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:38.047021 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0203 21:06:38.047466 (+ 445us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0203 21:06:38.047472 (+ 6us) client_negotiation.cc:769] Sending connection context
0203 21:06:38.047537 (+ 65us) client_negotiation.cc:240] Negotiation successful
0203 21:06:38.047601 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":126,"threads_started":1}
I20260203 21:06:38.047739 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:38.044287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:42817 (local address 127.0.102.129:34843)
0203 21:06:38.044410 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:38.044415 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:38.044768 (+ 353us) server_negotiation.cc:408] Connection header received
0203 21:06:38.044889 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:38.044893 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:38.044956 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:38.045040 (+ 84us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:06:38.045670 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:38.046267 (+ 597us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:38.047006 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:38.047223 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:38.047271 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:06:38.047357 (+ 86us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:06:38.047434 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:38.047556 (+ 122us) server_negotiation.cc:300] Negotiation successful
0203 21:06:38.047608 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260203 21:06:38.074515 1546 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:06:38.074695 1546 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:06:38.077787 1546 catalog_manager.cc:5941] T 00000000000000000000000000000000 P de4ddeb8e2fe47f2b27c416643b10684: Sending DeleteTablet for 3 replicas of tablet a551d7919f7741aca817ce78e04cbbfd
I20260203 21:06:38.078886 1827 tablet_service.cc:1552] Processing DeleteTablet for tablet a551d7919f7741aca817ce78e04cbbfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:38 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:39144
I20260203 21:06:38.079115 2080 tablet_replica.cc:333] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08: stopping tablet replica
I20260203 21:06:38.079247 2080 raft_consensus.cc:2243] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:06:38.079398 2080 raft_consensus.cc:2272] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:06:38.079767 1683 tablet_service.cc:1552] Processing DeleteTablet for tablet a551d7919f7741aca817ce78e04cbbfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:38 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:60710
I20260203 21:06:38.079982 2081 tablet_replica.cc:333] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6: stopping tablet replica
I20260203 21:06:38.080103 2081 raft_consensus.cc:2243] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [term 1 LEADER]: Raft consensus shutting down.
I20260203 21:06:38.080427 2081 raft_consensus.cc:2272] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:06:38.080473 2080 ts_tablet_manager.cc:1916] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:06:38.081349 2081 ts_tablet_manager.cc:1916] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:06:38.083551 1969 tablet_service.cc:1552] Processing DeleteTablet for tablet a551d7919f7741aca817ce78e04cbbfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:38 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:54726
I20260203 21:06:38.083765 2082 tablet_replica.cc:333] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c: stopping tablet replica
I20260203 21:06:38.083874 2082 raft_consensus.cc:2243] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:06:38.083999 2082 raft_consensus.cc:2272] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:06:38.084084 2081 ts_tablet_manager.cc:1929] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260203 21:06:38.084162 2081 log.cc:1199] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-0/wal/wals/a551d7919f7741aca817ce78e04cbbfd
I20260203 21:06:38.084419 2081 ts_tablet_manager.cc:1950] T a551d7919f7741aca817ce78e04cbbfd P 3cfa5e1ee9534cc58871e84df53170b6: Deleting consensus metadata
I20260203 21:06:38.084867 2082 ts_tablet_manager.cc:1916] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:06:38.085583 1531 catalog_manager.cc:4985] TS 3cfa5e1ee9534cc58871e84df53170b6 (127.0.102.129:34843): tablet a551d7919f7741aca817ce78e04cbbfd (table test-table [id=2c24bdc7e1ed462da4ef6ad06accf26b]) successfully deleted
I20260203 21:06:38.088063 2082 ts_tablet_manager.cc:1929] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260203 21:06:38.088165 2082 log.cc:1199] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-2/wal/wals/a551d7919f7741aca817ce78e04cbbfd
I20260203 21:06:38.088418 2082 ts_tablet_manager.cc:1950] T a551d7919f7741aca817ce78e04cbbfd P 48f49653806446e9b1363c915295951c: Deleting consensus metadata
I20260203 21:06:38.089593 1531 catalog_manager.cc:4985] TS 48f49653806446e9b1363c915295951c (127.0.102.131:38827): tablet a551d7919f7741aca817ce78e04cbbfd (table test-table [id=2c24bdc7e1ed462da4ef6ad06accf26b]) successfully deleted
I20260203 21:06:38.090003 2080 ts_tablet_manager.cc:1929] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260203 21:06:38.090097 2080 log.cc:1199] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1770152788143875-410-0/minicluster-data/ts-1/wal/wals/a551d7919f7741aca817ce78e04cbbfd
I20260203 21:06:38.090356 2080 ts_tablet_manager.cc:1950] T a551d7919f7741aca817ce78e04cbbfd P 3a05cb7f09c345918db000d4ca32dd08: Deleting consensus metadata
I20260203 21:06:38.091220 1531 catalog_manager.cc:4985] TS 3a05cb7f09c345918db000d4ca32dd08 (127.0.102.130:44321): tablet a551d7919f7741aca817ce78e04cbbfd (table test-table [id=2c24bdc7e1ed462da4ef6ad06accf26b]) successfully deleted
Feb 03 21:06:38 dist-test-slave-j889 krb5kdc[1498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.102.129@KRBTEST.COM
I20260203 21:06:38.102103 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:38.091479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.0.102.129:34843)
0203 21:06:38.091616 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:38.091620 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:38.091930 (+ 310us) server_negotiation.cc:408] Connection header received
0203 21:06:38.092057 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:38.092061 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:38.092120 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:38.092217 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:38.093285 (+ 1068us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:38.094045 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:38.094965 (+ 920us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:38.095172 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:38.097819 (+ 2647us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:38.097858 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:38.097886 (+ 28us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:38.097925 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:38.100024 (+ 2099us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:38.100555 (+ 531us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:38.100561 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:38.100563 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:38.100677 (+ 114us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:38.100914 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:38.100917 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:38.100919 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:38.101154 (+ 235us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:38.101406 (+ 252us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:38.101655 (+ 249us) server_negotiation.cc:300] Negotiation successful
0203 21:06:38.101792 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260203 21:06:38.102576 1745 server_base.cc:1142] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60728
I20260203 21:06:38.112902 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:38.104372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.0.102.190:37333)
0203 21:06:38.104504 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:38.104508 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:38.104960 (+ 452us) server_negotiation.cc:408] Connection header received
0203 21:06:38.105123 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:38.105136 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:38.105207 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:38.105324 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:38.106341 (+ 1017us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:38.107136 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:38.107975 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:38.108176 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:38.109475 (+ 1299us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:38.109500 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:38.109503 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:38.109537 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:38.111203 (+ 1666us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:38.111675 (+ 472us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:38.111679 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:38.111681 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:38.111733 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:38.111952 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:38.111955 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:38.111958 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:38.112151 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:38.112361 (+ 210us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:38.112568 (+ 207us) server_negotiation.cc:300] Negotiation successful
0203 21:06:38.112730 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260203 21:06:38.113745 1546 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58816
I20260203 21:06:38.115391 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 1620
I20260203 21:06:38.121770 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 1762
I20260203 21:06:38.128880 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 1904
I20260203 21:06:38.135459 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 1514
2026-02-03T21:06:38Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3879 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:06:38 dist-test-slave-j889 krb5kdc[2121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:06:38 dist-test-slave-j889 krb5kdc[2121](info): set up 2 sockets
Feb 03 21:06:38 dist-test-slave-j889 krb5kdc[2121](info): commencing operation
krb5kdc: starting...
W20260203 21:06:39.988659 410 mini_kdc.cc:121] Time spent starting KDC: real 1.831s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:06:40 dist-test-slave-j889 krb5kdc[2121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:06:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:06:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:40.124783 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:42201
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:42201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:40.228787 2137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:40.229031 2137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:40.229089 2137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:40.232452 2137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:06:40.232537 2137 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260203 21:06:40.232554 2137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:40.232575 2137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:40.232592 2137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:06:40.232607 2137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:06:40.237087 2137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:42201
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:42201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:40.238150 2137 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:40.239007 2137 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:40.244752 2143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:40.244863 2137 server_base.cc:1060] running on GCE node
W20260203 21:06:40.244746 2145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:40.244792 2142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:40.245545 2137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:40.246455 2137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:40.247632 2137 hybrid_clock.cc:648] HybridClock initialized: now 1770152800247612 us; error 34 us; skew 500 ppm
Feb 03 21:06:40 dist-test-slave-j889 krb5kdc[2121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:40.250460 2137 init.cc:377] Logged in from keytab as kudu/127.0.102.190@KRBTEST.COM (short username kudu)
I20260203 21:06:40.251562 2137 webserver.cc:492] Webserver started at http://127.0.102.190:41005/ using document root <none> and password file <none>
I20260203 21:06:40.252112 2137 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:40.252163 2137 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:40.252350 2137 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:40.254077 2137 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "bf2ff9d5092a44e8871418ca068b54b5"
format_stamp: "Formatted at 2026-02-03 21:06:40 on dist-test-slave-j889"
server_key: "4c656516b51e121235524eb45f5569ee"
server_key_iv: "8e7ce6946d768bc2b8c4f6848c05c858"
server_key_version: "encryptionkey@0"
I20260203 21:06:40.254549 2137 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "bf2ff9d5092a44e8871418ca068b54b5"
format_stamp: "Formatted at 2026-02-03 21:06:40 on dist-test-slave-j889"
server_key: "4c656516b51e121235524eb45f5569ee"
server_key_iv: "8e7ce6946d768bc2b8c4f6848c05c858"
server_key_version: "encryptionkey@0"
I20260203 21:06:40.258121 2137 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260203 21:06:40.260744 2152 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:40.261924 2137 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:06:40.262049 2137 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "bf2ff9d5092a44e8871418ca068b54b5"
format_stamp: "Formatted at 2026-02-03 21:06:40 on dist-test-slave-j889"
server_key: "4c656516b51e121235524eb45f5569ee"
server_key_iv: "8e7ce6946d768bc2b8c4f6848c05c858"
server_key_version: "encryptionkey@0"
I20260203 21:06:40.262151 2137 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:40.273994 2137 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:40.280412 2137 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:40.280618 2137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:40.288125 2214 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:42201 every 8 connection(s)
I20260203 21:06:40.288131 2137 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:42201
I20260203 21:06:40.289213 2137 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:06:40.290180 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 2137
I20260203 21:06:40.290299 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:06:40.290570 410 external_mini_cluster.cc:1468] Setting key 664f4f3c9f3438381f78649e757f43c4
I20260203 21:06:40.292841 2215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Feb 03 21:06:40 dist-test-slave-j889 krb5kdc[2121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:40.298506 2215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5: Bootstrap starting.
I20260203 21:06:40.300724 2215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:40.301389 2215 log.cc:826] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:40.302847 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:40.291722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.0.102.190:42201)
0203 21:06:40.292114 (+ 392us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:40.292123 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:40.292163 (+ 40us) server_negotiation.cc:408] Connection header received
0203 21:06:40.292994 (+ 831us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:40.293019 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:40.293370 (+ 351us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:40.293717 (+ 347us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:40.294439 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.295616 (+ 1177us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:40.296295 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.296573 (+ 278us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:40.298693 (+ 2120us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:40.298737 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:40.298747 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:40.298783 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:40.300447 (+ 1664us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:40.300910 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:40.300914 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:40.300919 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:40.300994 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:40.301207 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:40.301210 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:40.301212 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:40.301577 (+ 365us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:40.301767 (+ 190us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:40.302041 (+ 274us) server_negotiation.cc:300] Negotiation successful
0203 21:06:40.302274 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260203 21:06:40.303113 2215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5: No bootstrap required, opened a new log
I20260203 21:06:40.306190 2215 raft_consensus.cc:359] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf2ff9d5092a44e8871418ca068b54b5" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42201 } }
I20260203 21:06:40.306373 2215 raft_consensus.cc:385] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:40.306437 2215 raft_consensus.cc:740] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2ff9d5092a44e8871418ca068b54b5, State: Initialized, Role: FOLLOWER
I20260203 21:06:40.306828 2215 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf2ff9d5092a44e8871418ca068b54b5" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42201 } }
I20260203 21:06:40.306947 2215 raft_consensus.cc:399] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:06:40.307003 2215 raft_consensus.cc:493] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:06:40.307089 2215 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:40.307853 2215 raft_consensus.cc:515] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf2ff9d5092a44e8871418ca068b54b5" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42201 } }
I20260203 21:06:40.308172 2215 leader_election.cc:304] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf2ff9d5092a44e8871418ca068b54b5; no voters:
I20260203 21:06:40.308455 2215 leader_election.cc:290] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:06:40.308568 2220 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:40.308776 2220 raft_consensus.cc:697] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [term 1 LEADER]: Becoming Leader. State: Replica: bf2ff9d5092a44e8871418ca068b54b5, State: Running, Role: LEADER
I20260203 21:06:40.309062 2220 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf2ff9d5092a44e8871418ca068b54b5" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42201 } }
I20260203 21:06:40.309482 2215 sys_catalog.cc:565] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:06:40.310523 2222 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf2ff9d5092a44e8871418ca068b54b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf2ff9d5092a44e8871418ca068b54b5" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42201 } } }
I20260203 21:06:40.310626 2222 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [sys.catalog]: This master's current role is: LEADER
I20260203 21:06:40.310858 2221 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf2ff9d5092a44e8871418ca068b54b5. Latest consensus state: current_term: 1 leader_uuid: "bf2ff9d5092a44e8871418ca068b54b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf2ff9d5092a44e8871418ca068b54b5" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42201 } } }
I20260203 21:06:40.310937 2221 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5 [sys.catalog]: This master's current role is: LEADER
I20260203 21:06:40.311071 2229 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:06:40.314051 2229 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:06:40.319353 2229 catalog_manager.cc:1357] Generated new cluster ID: 0e525998884743618b20b7766382e5c9
I20260203 21:06:40.319434 2229 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:06:40.325580 2229 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:06:40.326634 2229 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:06:40.337396 2229 catalog_manager.cc:6027] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5: Generated new TSK 0
I20260203 21:06:40.338112 2229 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:40.389716 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:42201
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:40.501271 2243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:40.501509 2243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:40.501576 2243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:40.504957 2243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:40.505021 2243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:40.505075 2243 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260203 21:06:40.505093 2243 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260203 21:06:40.505154 2243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:06:40.509441 2243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.102.190:42201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:40.510535 2243 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:40.511411 2243 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:40.517956 2249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:40.517977 2248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:40.518198 2243 server_base.cc:1060] running on GCE node
W20260203 21:06:40.518397 2251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:40.518752 2243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:40.519268 2243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:40.520417 2243 hybrid_clock.cc:648] HybridClock initialized: now 1770152800520401 us; error 33 us; skew 500 ppm
Feb 03 21:06:40 dist-test-slave-j889 krb5kdc[2121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:40.522936 2243 init.cc:377] Logged in from keytab as kudu/127.0.102.129@KRBTEST.COM (short username kudu)
I20260203 21:06:40.523926 2243 webserver.cc:492] Webserver started at http://127.0.102.129:38881/ using document root <none> and password file <none>
I20260203 21:06:40.524456 2243 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:40.524507 2243 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:40.524736 2243 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:40.526388 2243 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "0e7d723a89424d3484fc057e3ab1a8e0"
format_stamp: "Formatted at 2026-02-03 21:06:40 on dist-test-slave-j889"
server_key: "39ff7e86a56ec3c957b67b33d19851f0"
server_key_iv: "8eabeae1991a7d0a41dd046615358746"
server_key_version: "encryptionkey@0"
I20260203 21:06:40.526837 2243 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "0e7d723a89424d3484fc057e3ab1a8e0"
format_stamp: "Formatted at 2026-02-03 21:06:40 on dist-test-slave-j889"
server_key: "39ff7e86a56ec3c957b67b33d19851f0"
server_key_iv: "8eabeae1991a7d0a41dd046615358746"
server_key_version: "encryptionkey@0"
I20260203 21:06:40.530167 2243 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260203 21:06:40.532459 2258 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:40.533584 2243 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:06:40.533690 2243 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "0e7d723a89424d3484fc057e3ab1a8e0"
format_stamp: "Formatted at 2026-02-03 21:06:40 on dist-test-slave-j889"
server_key: "39ff7e86a56ec3c957b67b33d19851f0"
server_key_iv: "8eabeae1991a7d0a41dd046615358746"
server_key_version: "encryptionkey@0"
I20260203 21:06:40.533787 2243 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:40.565579 2243 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:40.568463 2243 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:40.568663 2243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:40.570087 2243 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:40.570137 2243 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:40.570189 2243 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:40.570218 2243 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 03 21:06:40 dist-test-slave-j889 krb5kdc[2121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:40.581053 2243 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:37489
I20260203 21:06:40.581085 2373 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:37489 every 8 connection(s)
I20260203 21:06:40.582069 2243 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:06:40.582646 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:40.570764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:34201 (local address 127.0.102.190:42201)
0203 21:06:40.570876 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:40.570880 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:40.571695 (+ 815us) server_negotiation.cc:408] Connection header received
0203 21:06:40.572561 (+ 866us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:40.572564 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:40.572653 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:40.572732 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:40.574432 (+ 1700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.574992 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:40.575807 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.576014 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:40.578807 (+ 2793us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:40.578833 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:40.578835 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:40.578866 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:40.580399 (+ 1533us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:40.581002 (+ 603us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:40.581007 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:40.581009 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:40.581083 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:40.581466 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:40.581470 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:40.581472 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:40.581678 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:40.581783 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:40.582322 (+ 539us) server_negotiation.cc:300] Negotiation successful
0203 21:06:40.582484 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260203 21:06:40.583285 2267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:40.571112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42201 (local address 127.0.102.129:34201)
0203 21:06:40.571566 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:40.571601 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:40.572357 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:40.572933 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:40.572944 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:40.573503 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:40.574259 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:40.574290 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.575110 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:40.575115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:40.575671 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:40.575681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.575915 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:40.576554 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:40.576593 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:40.578588 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:40.580528 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:40.580534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:40.580546 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:40.580896 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:40.581257 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:40.581261 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:40.581262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:40.581354 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:40.581804 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:40.581809 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:40.582061 (+ 252us) client_negotiation.cc:769] Sending connection context
0203 21:06:40.582279 (+ 218us) client_negotiation.cc:240] Negotiation successful
0203 21:06:40.582545 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":149,"threads_started":1}
I20260203 21:06:40.584441 2374 heartbeater.cc:344] Connected to a master server at 127.0.102.190:42201
I20260203 21:06:40.584605 2374 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:40.585227 2374 heartbeater.cc:507] Master 127.0.102.190:42201 requested a full tablet report, sending...
I20260203 21:06:40.585556 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 2243
I20260203 21:06:40.585683 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:06:40.585928 410 external_mini_cluster.cc:1468] Setting key 13d554ac8f44e9e37d9c5119fbb27bda
I20260203 21:06:40.587605 2169 ts_manager.cc:194] Registered new tserver with Master: 0e7d723a89424d3484fc057e3ab1a8e0 (127.0.102.129:37489)
I20260203 21:06:40.588939 2169 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:34201
I20260203 21:06:40.594964 2380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:40.586675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42201 (local address 127.0.102.129:57181)
0203 21:06:40.587040 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:40.587053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:40.587149 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:40.587440 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:40.587443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:40.587639 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:40.587962 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:40.587971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.589143 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:40.589146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:40.589558 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:40.589568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.589681 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:40.591800 (+ 2119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:40.591814 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:40.592091 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:40.593827 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:40.593830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:40.593831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:40.593988 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:40.594206 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:40.594208 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:40.594209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:40.594244 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:40.594583 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:40.594585 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:40.594660 (+ 75us) client_negotiation.cc:769] Sending connection context
0203 21:06:40.594734 (+ 74us) client_negotiation.cc:240] Negotiation successful
0203 21:06:40.594829 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":93,"threads_started":1}
I20260203 21:06:40.595069 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:40.586666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:57181 (local address 127.0.102.190:42201)
0203 21:06:40.586916 (+ 250us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:40.586920 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:40.587123 (+ 203us) server_negotiation.cc:408] Connection header received
0203 21:06:40.587276 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:40.587279 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:40.587322 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:40.587401 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:40.588084 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.588981 (+ 897us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:40.589673 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.589837 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:40.592228 (+ 2391us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:40.592247 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:40.592250 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:40.592277 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:40.593724 (+ 1447us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:40.594075 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:40.594079 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:40.594080 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:40.594131 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:40.594313 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:40.594317 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:40.594319 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:40.594499 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:40.594600 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:40.594805 (+ 205us) server_negotiation.cc:300] Negotiation successful
0203 21:06:40.594919 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
WARNING: no policy specified for kudu/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:40.633798 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:42201
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:40.741204 2385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:40.741463 2385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:40.741539 2385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:40.744874 2385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:40.744948 2385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:40.745018 2385 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260203 21:06:40.745050 2385 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260203 21:06:40.745124 2385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:06:40.749374 2385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.102.190:42201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:40.750473 2385 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:40.751339 2385 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:40.758006 2393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:40.758071 2390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:40.758071 2391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:40.758432 2385 server_base.cc:1060] running on GCE node
I20260203 21:06:40.758823 2385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:40.759385 2385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:40.760567 2385 hybrid_clock.cc:648] HybridClock initialized: now 1770152800760544 us; error 37 us; skew 500 ppm
Feb 03 21:06:40 dist-test-slave-j889 krb5kdc[2121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:40.763377 2385 init.cc:377] Logged in from keytab as kudu/127.0.102.130@KRBTEST.COM (short username kudu)
I20260203 21:06:40.764511 2385 webserver.cc:492] Webserver started at http://127.0.102.130:42877/ using document root <none> and password file <none>
I20260203 21:06:40.765100 2385 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:40.765153 2385 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:40.765326 2385 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:40.767148 2385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "730c029960fb4e2ba1d6b03c82f399a9"
format_stamp: "Formatted at 2026-02-03 21:06:40 on dist-test-slave-j889"
server_key: "fd1b9c04a1232ff0919a6d32b1442166"
server_key_iv: "54f84ac5f6fc23f50de5bf7f704cb6d2"
server_key_version: "encryptionkey@0"
I20260203 21:06:40.767616 2385 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "730c029960fb4e2ba1d6b03c82f399a9"
format_stamp: "Formatted at 2026-02-03 21:06:40 on dist-test-slave-j889"
server_key: "fd1b9c04a1232ff0919a6d32b1442166"
server_key_iv: "54f84ac5f6fc23f50de5bf7f704cb6d2"
server_key_version: "encryptionkey@0"
I20260203 21:06:40.771138 2385 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260203 21:06:40.773566 2400 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:40.774713 2385 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260203 21:06:40.774828 2385 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "730c029960fb4e2ba1d6b03c82f399a9"
format_stamp: "Formatted at 2026-02-03 21:06:40 on dist-test-slave-j889"
server_key: "fd1b9c04a1232ff0919a6d32b1442166"
server_key_iv: "54f84ac5f6fc23f50de5bf7f704cb6d2"
server_key_version: "encryptionkey@0"
I20260203 21:06:40.774930 2385 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:40.791110 2385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:40.793673 2385 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:40.793828 2385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:40.795251 2385 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:40.795302 2385 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:40.795351 2385 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:40.795378 2385 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 03 21:06:40 dist-test-slave-j889 krb5kdc[2121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:40.805850 2385 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:46385
I20260203 21:06:40.805884 2515 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:46385 every 8 connection(s)
I20260203 21:06:40.806864 2385 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:06:40.807571 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:40.795846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:36183 (local address 127.0.102.190:42201)
0203 21:06:40.795981 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:40.795986 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:40.796771 (+ 785us) server_negotiation.cc:408] Connection header received
0203 21:06:40.797832 (+ 1061us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:40.797835 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:40.797884 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:40.797973 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:40.799756 (+ 1783us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.800376 (+ 620us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:40.801302 (+ 926us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.801508 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:40.804105 (+ 2597us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:40.804130 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:40.804134 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:40.804167 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:40.805634 (+ 1467us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:40.806103 (+ 469us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:40.806106 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:40.806108 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:40.806164 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:40.806456 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:40.806458 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:40.806459 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:40.806623 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:40.806693 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:40.807291 (+ 598us) server_negotiation.cc:300] Negotiation successful
0203 21:06:40.807422 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260203 21:06:40.808204 2409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:40.796121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42201 (local address 127.0.102.130:36183)
0203 21:06:40.796601 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:40.796674 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:40.797617 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:40.798166 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:40.798180 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:40.798756 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:40.799523 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:40.799542 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.800491 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:40.800495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:40.801145 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:40.801156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.801392 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:40.802035 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:40.802070 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:40.803938 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:40.805734 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:40.805740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:40.805752 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:40.806011 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:40.806262 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:40.806265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:40.806267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:40.806369 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:40.806716 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:40.806722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:40.807022 (+ 300us) client_negotiation.cc:769] Sending connection context
0203 21:06:40.807251 (+ 229us) client_negotiation.cc:240] Negotiation successful
0203 21:06:40.807535 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":115,"threads_started":1}
I20260203 21:06:40.809199 2516 heartbeater.cc:344] Connected to a master server at 127.0.102.190:42201
I20260203 21:06:40.809204 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 2385
I20260203 21:06:40.809325 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:06:40.809352 2516 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:40.809593 410 external_mini_cluster.cc:1468] Setting key d731b62e8b0905dabbb047189b6e0b4c
I20260203 21:06:40.809886 2516 heartbeater.cc:507] Master 127.0.102.190:42201 requested a full tablet report, sending...
I20260203 21:06:40.811340 2168 ts_manager.cc:194] Registered new tserver with Master: 730c029960fb4e2ba1d6b03c82f399a9 (127.0.102.130:46385)
I20260203 21:06:40.811903 2168 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.130@KRBTEST.COM'} at 127.0.102.130:36183
WARNING: no policy specified for kudu/127.0.102.131@KRBTEST.COM; defaulting to no policy
I20260203 21:06:40.818353 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:40.811393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:44617 (local address 127.0.102.190:42201)
0203 21:06:40.811498 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:40.811502 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:40.812395 (+ 893us) server_negotiation.cc:408] Connection header received
0203 21:06:40.812514 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:40.812518 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:40.812564 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:40.812669 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:40.813214 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.813910 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:40.814529 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.814709 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:40.815559 (+ 850us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:40.815575 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:40.815577 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:40.815599 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:40.816947 (+ 1348us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:40.817313 (+ 366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:40.817316 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:40.817318 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:40.817365 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:40.817566 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:40.817570 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:40.817572 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:40.817746 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:40.817830 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:40.818101 (+ 271us) server_negotiation.cc:300] Negotiation successful
0203 21:06:40.818205 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260203 21:06:40.818373 2522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:40.811334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42201 (local address 127.0.102.130:44617)
0203 21:06:40.812298 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:40.812313 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:40.812421 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:40.812706 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:40.812709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:40.812894 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:40.813104 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:40.813110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.814010 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:40.814013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:40.814429 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:40.814437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:40.814536 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:40.815129 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:40.815141 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:40.815429 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:40.817048 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:40.817052 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:40.817054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:40.817218 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:40.817442 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:40.817445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:40.817446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:40.817489 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:40.817872 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:40.817874 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:40.817951 (+ 77us) client_negotiation.cc:769] Sending connection context
0203 21:06:40.818038 (+ 87us) client_negotiation.cc:240] Negotiation successful
0203 21:06:40.818169 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":80,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:40.854022 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:42201
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:40.957103 2527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:40.957329 2527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:40.957425 2527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:40.960793 2527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:40.960870 2527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:40.960947 2527 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260203 21:06:40.960971 2527 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260203 21:06:40.961030 2527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:06:40.965480 2527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.102.190:42201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:40.966619 2527 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:40.967485 2527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:40.974022 2535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:40.973963 2533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:40.973965 2532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:40.974193 2527 server_base.cc:1060] running on GCE node
I20260203 21:06:40.974669 2527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:40.975184 2527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:40.976341 2527 hybrid_clock.cc:648] HybridClock initialized: now 1770152800976328 us; error 30 us; skew 500 ppm
Feb 03 21:06:40 dist-test-slave-j889 krb5kdc[2121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:40.978938 2527 init.cc:377] Logged in from keytab as kudu/127.0.102.131@KRBTEST.COM (short username kudu)
I20260203 21:06:40.979933 2527 webserver.cc:492] Webserver started at http://127.0.102.131:34521/ using document root <none> and password file <none>
I20260203 21:06:40.980440 2527 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:40.980485 2527 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:40.980710 2527 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:40.982357 2527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "8c3cd322b4244c5ebffbec1606b2d5be"
format_stamp: "Formatted at 2026-02-03 21:06:40 on dist-test-slave-j889"
server_key: "405ab971f6318ace453bc0da8a17ec7f"
server_key_iv: "185368a0f1ed9caf24f290eea10b95d6"
server_key_version: "encryptionkey@0"
I20260203 21:06:40.982788 2527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "8c3cd322b4244c5ebffbec1606b2d5be"
format_stamp: "Formatted at 2026-02-03 21:06:40 on dist-test-slave-j889"
server_key: "405ab971f6318ace453bc0da8a17ec7f"
server_key_iv: "185368a0f1ed9caf24f290eea10b95d6"
server_key_version: "encryptionkey@0"
I20260203 21:06:40.985984 2527 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260203 21:06:40.988050 2542 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:40.989079 2527 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:06:40.989164 2527 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "8c3cd322b4244c5ebffbec1606b2d5be"
format_stamp: "Formatted at 2026-02-03 21:06:40 on dist-test-slave-j889"
server_key: "405ab971f6318ace453bc0da8a17ec7f"
server_key_iv: "185368a0f1ed9caf24f290eea10b95d6"
server_key_version: "encryptionkey@0"
I20260203 21:06:40.989253 2527 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:41.013556 2527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:41.016029 2527 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:41.016185 2527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:41.017663 2527 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:41.017712 2527 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:41.017758 2527 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:41.017792 2527 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 03 21:06:41 dist-test-slave-j889 krb5kdc[2121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:41.028748 2527 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:39909
I20260203 21:06:41.028878 2657 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:39909 every 8 connection(s)
I20260203 21:06:41.029827 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.018226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:46241 (local address 127.0.102.190:42201)
0203 21:06:41.018356 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:41.018360 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:41.019259 (+ 899us) server_negotiation.cc:408] Connection header received
0203 21:06:41.020134 (+ 875us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:41.020138 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:41.020198 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:41.020273 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:41.021732 (+ 1459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.022338 (+ 606us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.022998 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.023177 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.025657 (+ 2480us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:41.025678 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:41.025681 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:41.025710 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:41.027552 (+ 1842us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:41.028166 (+ 614us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:41.028169 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:41.028171 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:41.028222 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:41.028589 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:41.028593 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:41.028595 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:41.028833 (+ 238us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:41.028936 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:41.029580 (+ 644us) server_negotiation.cc:300] Negotiation successful
0203 21:06:41.029703 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260203 21:06:41.029970 2527 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
I20260203 21:06:41.030625 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.018589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42201 (local address 127.0.102.131:46241)
0203 21:06:41.019116 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:41.019163 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:41.019937 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:41.020478 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:41.020488 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:41.021011 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:41.021580 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.021593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.022459 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.022462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:41.022885 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.022892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.023100 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.023719 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:41.023752 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:41.025454 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:41.027681 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:41.027686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:41.027698 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:41.028024 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:41.028369 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:41.028371 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:41.028373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:41.028494 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:41.028953 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:41.028959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:41.029264 (+ 305us) client_negotiation.cc:769] Sending connection context
0203 21:06:41.029499 (+ 235us) client_negotiation.cc:240] Negotiation successful
0203 21:06:41.029791 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":139,"threads_started":1}
I20260203 21:06:41.031958 2658 heartbeater.cc:344] Connected to a master server at 127.0.102.190:42201
I20260203 21:06:41.032112 2658 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:41.032624 2658 heartbeater.cc:507] Master 127.0.102.190:42201 requested a full tablet report, sending...
I20260203 21:06:41.033653 2169 ts_manager.cc:194] Registered new tserver with Master: 8c3cd322b4244c5ebffbec1606b2d5be (127.0.102.131:39909)
I20260203 21:06:41.034242 2169 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.131@KRBTEST.COM'} at 127.0.102.131:46241
I20260203 21:06:41.039623 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 2527
I20260203 21:06:41.039749 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:06:41.040006 410 external_mini_cluster.cc:1468] Setting key 6a70935bdc1ba0e46f11eaf0a03dc655
I20260203 21:06:41.040469 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.034134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42201 (local address 127.0.102.131:43647)
0203 21:06:41.034402 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:41.034418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:41.034506 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:41.034723 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:41.034726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:41.034923 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:41.035188 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.035197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.035913 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.035917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:41.036412 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.036421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.036511 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.037012 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:41.037025 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:41.037361 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:41.038959 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:41.038962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:41.038964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:41.039158 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:41.039483 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:41.039487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:41.039489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:41.039551 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:41.039966 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:41.039969 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:41.040067 (+ 98us) client_negotiation.cc:769] Sending connection context
0203 21:06:41.040177 (+ 110us) client_negotiation.cc:240] Negotiation successful
0203 21:06:41.040319 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
I20260203 21:06:41.040658 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.034234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:43647 (local address 127.0.102.190:42201)
0203 21:06:41.034547 (+ 313us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:41.034551 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:41.034564 (+ 13us) server_negotiation.cc:408] Connection header received
0203 21:06:41.034603 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:41.034605 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:41.034638 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:41.034698 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:41.035317 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.035808 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.036529 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.036745 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.037479 (+ 734us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:41.037495 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:41.037498 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:41.037520 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:41.038849 (+ 1329us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:41.039305 (+ 456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:41.039308 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:41.039310 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:41.039361 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:41.039683 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:41.039686 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:41.039687 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:41.039874 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:41.039955 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:41.040317 (+ 362us) server_negotiation.cc:300] Negotiation successful
0203 21:06:41.040454 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260203 21:06:41.042351 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 03 21:06:41 dist-test-slave-j889 krb5kdc[2121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 03 21:06:41 dist-test-slave-j889 krb5kdc[2121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:41.065114 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.057771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55558 (local address 127.0.102.190:42201)
0203 21:06:41.057929 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:41.057933 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:41.057944 (+ 11us) server_negotiation.cc:408] Connection header received
0203 21:06:41.057983 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:41.057986 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:41.058030 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:41.058099 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:41.058903 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.059408 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.060071 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.060257 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.062375 (+ 2118us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:41.062396 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:41.062398 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:41.062427 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:41.063728 (+ 1301us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:41.064091 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:41.064094 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:41.064095 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:41.064142 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:41.064340 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:41.064344 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:41.064345 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:41.064498 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:41.064618 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:41.064843 (+ 225us) server_negotiation.cc:300] Negotiation successful
0203 21:06:41.064957 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260203 21:06:41.067835 2169 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260203 21:06:41.070199 2169 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:06:41.082633 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.078057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:46385 (local address 127.0.0.1:47128)
0203 21:06:41.078678 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:41.078719 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:41.078841 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:41.079877 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:41.079880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:41.079895 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:41.080233 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.080242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.081279 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.081282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:41.082112 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.082120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.082217 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.082270 (+ 53us) client_negotiation.cc:769] Sending connection context
0203 21:06:41.082358 (+ 88us) client_negotiation.cc:240] Negotiation successful
0203 21:06:41.082447 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":66,"threads_started":1}
I20260203 21:06:41.083086 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.078221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.0.102.130:46385)
0203 21:06:41.079351 (+ 1130us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:41.079357 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:41.079375 (+ 18us) server_negotiation.cc:408] Connection header received
0203 21:06:41.079450 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:41.079455 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:41.079702 (+ 247us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:41.079879 (+ 177us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:41.080387 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.081176 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.082237 (+ 1061us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.082755 (+ 518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.082863 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:41.082918 (+ 55us) server_negotiation.cc:300] Negotiation successful
0203 21:06:41.082982 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1039,"thread_start_us":114,"threads_started":1}
I20260203 21:06:41.083518 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.078198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:37489 (local address 127.0.0.1:36132)
0203 21:06:41.079055 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:41.079067 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:41.079136 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:41.079824 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:41.079830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:41.079853 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:41.080233 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.080242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.081902 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.081906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:41.082980 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.082991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.083091 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.083112 (+ 21us) client_negotiation.cc:769] Sending connection context
0203 21:06:41.083315 (+ 203us) client_negotiation.cc:240] Negotiation successful
0203 21:06:41.083379 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":128,"threads_started":1}
I20260203 21:06:41.084025 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.078795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:39909 (local address 127.0.0.1:44326)
0203 21:06:41.079609 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:41.079624 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:41.079731 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:41.080115 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:41.080118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:41.080146 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:41.080404 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.080413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.081902 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.081906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:41.083685 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.083695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.083797 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.083813 (+ 16us) client_negotiation.cc:769] Sending connection context
0203 21:06:41.083859 (+ 46us) client_negotiation.cc:240] Negotiation successful
0203 21:06:41.083915 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":96,"threads_started":1}
I20260203 21:06:41.084986 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.078941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36132 (local address 127.0.102.129:37489)
0203 21:06:41.079283 (+ 342us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:41.079291 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:41.079317 (+ 26us) server_negotiation.cc:408] Connection header received
0203 21:06:41.079407 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:41.079412 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:41.079632 (+ 220us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:41.080391 (+ 759us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:41.080561 (+ 170us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.081758 (+ 1197us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.083134 (+ 1376us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.083674 (+ 540us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.084718 (+ 1044us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:41.084784 (+ 66us) server_negotiation.cc:300] Negotiation successful
0203 21:06:41.084869 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":94,"threads_started":1}
I20260203 21:06:41.085017 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.079251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.0.102.131:39909)
0203 21:06:41.079576 (+ 325us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:41.079582 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:41.079702 (+ 120us) server_negotiation.cc:408] Connection header received
0203 21:06:41.079848 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:41.079853 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:41.079992 (+ 139us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:41.080116 (+ 124us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:41.080532 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.081766 (+ 1234us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.084196 (+ 2430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.084672 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.084772 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:41.084824 (+ 52us) server_negotiation.cc:300] Negotiation successful
0203 21:06:41.084900 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":126,"threads_started":1}
I20260203 21:06:41.086043 2450 tablet_service.cc:1505] Processing CreateTablet for tablet e3d9f628bd144652a8d235734aa25c1c (DEFAULT_TABLE table=test-table [id=760e650c04444e41b0f2a04c222b472b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:41.086935 2450 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3d9f628bd144652a8d235734aa25c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:41.087460 2592 tablet_service.cc:1505] Processing CreateTablet for tablet e3d9f628bd144652a8d235734aa25c1c (DEFAULT_TABLE table=test-table [id=760e650c04444e41b0f2a04c222b472b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:41.088205 2592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3d9f628bd144652a8d235734aa25c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:41.091331 2308 tablet_service.cc:1505] Processing CreateTablet for tablet e3d9f628bd144652a8d235734aa25c1c (DEFAULT_TABLE table=test-table [id=760e650c04444e41b0f2a04c222b472b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:41.092247 2308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3d9f628bd144652a8d235734aa25c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:41.092429 2681 tablet_bootstrap.cc:492] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9: Bootstrap starting.
I20260203 21:06:41.094008 2682 tablet_bootstrap.cc:492] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be: Bootstrap starting.
I20260203 21:06:41.095867 2681 tablet_bootstrap.cc:654] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:41.096000 2682 tablet_bootstrap.cc:654] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:41.097208 2681 log.cc:826] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:41.097208 2682 log.cc:826] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:41.097426 2683 tablet_bootstrap.cc:492] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0: Bootstrap starting.
I20260203 21:06:41.098912 2682 tablet_bootstrap.cc:492] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be: No bootstrap required, opened a new log
I20260203 21:06:41.099099 2682 ts_tablet_manager.cc:1403] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260203 21:06:41.099326 2681 tablet_bootstrap.cc:492] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9: No bootstrap required, opened a new log
I20260203 21:06:41.099332 2683 tablet_bootstrap.cc:654] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:41.099511 2681 ts_tablet_manager.cc:1403] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260203 21:06:41.100090 2683 log.cc:826] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:41.101907 2683 tablet_bootstrap.cc:492] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0: No bootstrap required, opened a new log
I20260203 21:06:41.102098 2683 ts_tablet_manager.cc:1403] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260203 21:06:41.102110 2681 raft_consensus.cc:359] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.102169 2682 raft_consensus.cc:359] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.102305 2681 raft_consensus.cc:385] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:41.102363 2681 raft_consensus.cc:740] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730c029960fb4e2ba1d6b03c82f399a9, State: Initialized, Role: FOLLOWER
I20260203 21:06:41.102408 2682 raft_consensus.cc:385] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:41.102475 2682 raft_consensus.cc:740] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3cd322b4244c5ebffbec1606b2d5be, State: Initialized, Role: FOLLOWER
I20260203 21:06:41.102815 2681 consensus_queue.cc:260] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.102882 2682 consensus_queue.cc:260] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.103430 2516 heartbeater.cc:499] Master 127.0.102.190:42201 was elected leader, sending a full tablet report...
I20260203 21:06:41.103621 2681 ts_tablet_manager.cc:1434] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20260203 21:06:41.103771 2682 ts_tablet_manager.cc:1434] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260203 21:06:41.104126 2658 heartbeater.cc:499] Master 127.0.102.190:42201 was elected leader, sending a full tablet report...
I20260203 21:06:41.105154 2683 raft_consensus.cc:359] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.105415 2683 raft_consensus.cc:385] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:41.105516 2683 raft_consensus.cc:740] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7d723a89424d3484fc057e3ab1a8e0, State: Initialized, Role: FOLLOWER
I20260203 21:06:41.105926 2683 consensus_queue.cc:260] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.106586 2374 heartbeater.cc:499] Master 127.0.102.190:42201 was elected leader, sending a full tablet report...
I20260203 21:06:41.107033 2683 ts_tablet_manager.cc:1434] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260203 21:06:41.280979 2659 tablet.cc:2378] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260203 21:06:41.308230 2517 tablet.cc:2378] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260203 21:06:41.333758 2375 tablet.cc:2378] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260203 21:06:41.350378 2688 raft_consensus.cc:493] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:06:41.350569 2688 raft_consensus.cc:515] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.351624 2688 leader_election.cc:290] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 730c029960fb4e2ba1d6b03c82f399a9 (127.0.102.130:46385), 0e7d723a89424d3484fc057e3ab1a8e0 (127.0.102.129:37489)
I20260203 21:06:41.354521 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.351898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:37489 (local address 127.0.102.131:51299)
0203 21:06:41.352015 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:41.352030 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:41.352125 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:41.352378 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:41.352381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:41.352410 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:41.352697 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.352706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.353539 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.353542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:41.354180 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.354189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.354282 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.354300 (+ 18us) client_negotiation.cc:769] Sending connection context
0203 21:06:41.354333 (+ 33us) client_negotiation.cc:240] Negotiation successful
0203 21:06:41.354379 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260203 21:06:41.354714 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.351898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:46385 (local address 127.0.102.131:36121)
0203 21:06:41.352365 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:41.352389 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:41.352469 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:41.352709 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:41.352711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:41.352724 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:41.353014 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.353021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.353746 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.353749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:41.354326 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.354334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.354449 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.354466 (+ 17us) client_negotiation.cc:769] Sending connection context
0203 21:06:41.354517 (+ 51us) client_negotiation.cc:240] Negotiation successful
0203 21:06:41.354570 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":47,"thread_start_us":109,"threads_started":1}
I20260203 21:06:41.355021 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.352000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:51299 (local address 127.0.102.129:37489)
0203 21:06:41.352128 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:41.352133 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:41.352149 (+ 16us) server_negotiation.cc:408] Connection header received
0203 21:06:41.352201 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:41.352220 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:41.352270 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:41.352346 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:41.352816 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.353421 (+ 605us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.354298 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.354759 (+ 461us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.354801 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:41.354854 (+ 53us) server_negotiation.cc:300] Negotiation successful
0203 21:06:41.354912 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260203 21:06:41.355211 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.352145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:36121 (local address 127.0.102.130:46385)
0203 21:06:41.352295 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:41.352299 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:41.352453 (+ 154us) server_negotiation.cc:408] Connection header received
0203 21:06:41.352542 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:41.352545 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:41.352592 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:41.352705 (+ 113us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:41.353119 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.353628 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.354466 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.354947 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.354984 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:41.355035 (+ 51us) server_negotiation.cc:300] Negotiation successful
0203 21:06:41.355096 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260203 21:06:41.355587 2326 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3d9f628bd144652a8d235734aa25c1c" candidate_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" is_pre_election: true
I20260203 21:06:41.355760 2470 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3d9f628bd144652a8d235734aa25c1c" candidate_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730c029960fb4e2ba1d6b03c82f399a9" is_pre_election: true
I20260203 21:06:41.355849 2326 raft_consensus.cc:2468] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c3cd322b4244c5ebffbec1606b2d5be in term 0.
I20260203 21:06:41.356001 2470 raft_consensus.cc:2468] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c3cd322b4244c5ebffbec1606b2d5be in term 0.
I20260203 21:06:41.356348 2545 leader_election.cc:304] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e7d723a89424d3484fc057e3ab1a8e0, 730c029960fb4e2ba1d6b03c82f399a9, 8c3cd322b4244c5ebffbec1606b2d5be; no voters:
I20260203 21:06:41.356611 2688 raft_consensus.cc:2804] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:06:41.356710 2688 raft_consensus.cc:493] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:06:41.356750 2688 raft_consensus.cc:3060] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:41.357769 2688 raft_consensus.cc:515] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.358131 2688 leader_election.cc:290] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [CANDIDATE]: Term 1 election: Requested vote from peers 730c029960fb4e2ba1d6b03c82f399a9 (127.0.102.130:46385), 0e7d723a89424d3484fc057e3ab1a8e0 (127.0.102.129:37489)
I20260203 21:06:41.358494 2326 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3d9f628bd144652a8d235734aa25c1c" candidate_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e7d723a89424d3484fc057e3ab1a8e0"
I20260203 21:06:41.358530 2470 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3d9f628bd144652a8d235734aa25c1c" candidate_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730c029960fb4e2ba1d6b03c82f399a9"
I20260203 21:06:41.358608 2326 raft_consensus.cc:3060] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:41.358670 2470 raft_consensus.cc:3060] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:41.359632 2326 raft_consensus.cc:2468] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3cd322b4244c5ebffbec1606b2d5be in term 1.
I20260203 21:06:41.359686 2470 raft_consensus.cc:2468] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3cd322b4244c5ebffbec1606b2d5be in term 1.
I20260203 21:06:41.359946 2545 leader_election.cc:304] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e7d723a89424d3484fc057e3ab1a8e0, 8c3cd322b4244c5ebffbec1606b2d5be; no voters:
I20260203 21:06:41.360116 2688 raft_consensus.cc:2804] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:41.360301 2688 raft_consensus.cc:697] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 LEADER]: Becoming Leader. State: Replica: 8c3cd322b4244c5ebffbec1606b2d5be, State: Running, Role: LEADER
I20260203 21:06:41.360620 2688 consensus_queue.cc:237] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.363971 2169 catalog_manager.cc:5654] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c3cd322b4244c5ebffbec1606b2d5be (127.0.102.131). New cstate: current_term: 1 leader_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } health_report { overall_health: HEALTHY } } }
Feb 03 21:06:41 dist-test-slave-j889 krb5kdc[2121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:41.405699 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.398412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42201 (local address 127.0.0.1:55572)
0203 21:06:41.398719 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:41.398734 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:41.398846 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:41.399053 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:41.399056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:41.399503 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:41.399698 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.399706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.400384 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.400386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:41.400783 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.400791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.400888 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.401415 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:41.401428 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:41.402786 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:41.404195 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:41.404203 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:41.404208 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:41.404552 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:41.404817 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:41.404820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:41.404822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:41.404869 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:41.405172 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:41.405179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:41.405272 (+ 93us) client_negotiation.cc:769] Sending connection context
0203 21:06:41.405369 (+ 97us) client_negotiation.cc:240] Negotiation successful
0203 21:06:41.405511 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":116,"threads_started":1}
I20260203 21:06:41.405699 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.398528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.0.102.190:42201)
0203 21:06:41.398695 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:41.398700 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:41.398799 (+ 99us) server_negotiation.cc:408] Connection header received
0203 21:06:41.398924 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:41.398927 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:41.398966 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:41.399026 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:41.399807 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.400284 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.400876 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.401032 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.402890 (+ 1858us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:41.402904 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:41.402906 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:41.402925 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:41.404085 (+ 1160us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:41.404695 (+ 610us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:41.404698 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:41.404700 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:41.404740 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:41.404939 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:41.404942 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:41.404945 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:41.405079 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:41.405160 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:41.405431 (+ 271us) server_negotiation.cc:300] Negotiation successful
0203 21:06:41.405562 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":55}
I20260203 21:06:41.409528 2169 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:55572:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260203 21:06:41.410269 2169 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:06:41.414980 2308 tablet_service.cc:1505] Processing CreateTablet for tablet 07601e39784e4bf69a9737e75bc03dbf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3683657a11904e8daf05a2947420cff2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260203 21:06:41.415122 2450 tablet_service.cc:1505] Processing CreateTablet for tablet 07601e39784e4bf69a9737e75bc03dbf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3683657a11904e8daf05a2947420cff2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260203 21:06:41.415664 2308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07601e39784e4bf69a9737e75bc03dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:41.415664 2450 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07601e39784e4bf69a9737e75bc03dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:41.417017 2592 tablet_service.cc:1505] Processing CreateTablet for tablet 07601e39784e4bf69a9737e75bc03dbf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3683657a11904e8daf05a2947420cff2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260203 21:06:41.417654 2592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07601e39784e4bf69a9737e75bc03dbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:41.418654 2683 tablet_bootstrap.cc:492] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0: Bootstrap starting.
I20260203 21:06:41.418916 2681 tablet_bootstrap.cc:492] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9: Bootstrap starting.
I20260203 21:06:41.419688 2683 tablet_bootstrap.cc:654] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:41.419734 2681 tablet_bootstrap.cc:654] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:41.420130 2682 tablet_bootstrap.cc:492] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be: Bootstrap starting.
I20260203 21:06:41.421190 2683 tablet_bootstrap.cc:492] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0: No bootstrap required, opened a new log
I20260203 21:06:41.421195 2682 tablet_bootstrap.cc:654] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:41.421271 2683 ts_tablet_manager.cc:1403] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260203 21:06:41.421443 2681 tablet_bootstrap.cc:492] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9: No bootstrap required, opened a new log
I20260203 21:06:41.421551 2681 ts_tablet_manager.cc:1403] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260203 21:06:41.421736 2683 raft_consensus.cc:359] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.421828 2683 raft_consensus.cc:385] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:41.421862 2683 raft_consensus.cc:740] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7d723a89424d3484fc057e3ab1a8e0, State: Initialized, Role: FOLLOWER
I20260203 21:06:41.421978 2683 consensus_queue.cc:260] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.422032 2681 raft_consensus.cc:359] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.422152 2681 raft_consensus.cc:385] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:41.422197 2681 raft_consensus.cc:740] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730c029960fb4e2ba1d6b03c82f399a9, State: Initialized, Role: FOLLOWER
I20260203 21:06:41.422226 2683 ts_tablet_manager.cc:1434] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:06:41.422338 2681 consensus_queue.cc:260] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.422452 2689 tablet_replica.cc:442] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } } }
I20260203 21:06:41.422555 2689 tablet_replica.cc:445] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:06:41.422654 2682 tablet_bootstrap.cc:492] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be: No bootstrap required, opened a new log
I20260203 21:06:41.422600 2687 tablet_replica.cc:442] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } } }
I20260203 21:06:41.422721 2682 ts_tablet_manager.cc:1403] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260203 21:06:41.422724 2687 tablet_replica.cc:445] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:06:41.422603 2681 ts_tablet_manager.cc:1434] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260203 21:06:41.423125 2682 raft_consensus.cc:359] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.423218 2682 raft_consensus.cc:385] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:41.423246 2682 raft_consensus.cc:740] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3cd322b4244c5ebffbec1606b2d5be, State: Initialized, Role: FOLLOWER
I20260203 21:06:41.423353 2682 consensus_queue.cc:260] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.423592 2682 ts_tablet_manager.cc:1434] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260203 21:06:41.423625 2688 tablet_replica.cc:442] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } } }
I20260203 21:06:41.423735 2688 tablet_replica.cc:445] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:06:41.537855 2687 raft_consensus.cc:493] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:06:41.538028 2687 raft_consensus.cc:515] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.539285 2687 leader_election.cc:290] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e7d723a89424d3484fc057e3ab1a8e0 (127.0.102.129:37489), 8c3cd322b4244c5ebffbec1606b2d5be (127.0.102.131:39909)
I20260203 21:06:41.542304 2703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.539534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:39909 (local address 127.0.102.130:35443)
0203 21:06:41.539947 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:41.539960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:41.540052 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:41.540286 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:41.540289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:41.540305 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:41.540528 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.540534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.541297 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.541301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:41.541918 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.541926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.542023 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.542045 (+ 22us) client_negotiation.cc:769] Sending connection context
0203 21:06:41.542093 (+ 48us) client_negotiation.cc:240] Negotiation successful
0203 21:06:41.542148 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":101,"threads_started":1}
I20260203 21:06:41.542322 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.539473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:37489 (local address 127.0.102.130:56983)
0203 21:06:41.539859 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:41.539873 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:41.540024 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:41.540268 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:41.540271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:41.540287 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:06:41.540585 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.540597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.541323 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.541326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:41.541926 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.541934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.542037 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.542056 (+ 19us) client_negotiation.cc:769] Sending connection context
0203 21:06:41.542103 (+ 47us) client_negotiation.cc:240] Negotiation successful
0203 21:06:41.542155 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"spinlock_wait_cycles":20224,"thread_start_us":92,"threads_started":1}
I20260203 21:06:41.542723 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.539603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:56983 (local address 127.0.102.129:37489)
0203 21:06:41.539731 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:41.539736 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:41.539945 (+ 209us) server_negotiation.cc:408] Connection header received
0203 21:06:41.540126 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:41.540130 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:41.540179 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:41.540233 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:41.540724 (+ 491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.541217 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.542040 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.542479 (+ 439us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.542520 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:41.542568 (+ 48us) server_negotiation.cc:300] Negotiation successful
0203 21:06:41.542618 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260203 21:06:41.542742 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.539686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:35443 (local address 127.0.102.131:39909)
0203 21:06:41.539816 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:41.539820 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:41.540028 (+ 208us) server_negotiation.cc:408] Connection header received
0203 21:06:41.540151 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:41.540154 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:41.540195 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:41.540262 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:06:41.540699 (+ 437us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.541193 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.542037 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.542483 (+ 446us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.542520 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:41.542567 (+ 47us) server_negotiation.cc:300] Negotiation successful
0203 21:06:41.542624 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260203 21:06:41.543124 2326 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07601e39784e4bf69a9737e75bc03dbf" candidate_uuid: "730c029960fb4e2ba1d6b03c82f399a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" is_pre_election: true
I20260203 21:06:41.543236 2612 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07601e39784e4bf69a9737e75bc03dbf" candidate_uuid: "730c029960fb4e2ba1d6b03c82f399a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" is_pre_election: true
I20260203 21:06:41.543270 2326 raft_consensus.cc:2468] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730c029960fb4e2ba1d6b03c82f399a9 in term 0.
I20260203 21:06:41.543417 2612 raft_consensus.cc:2468] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730c029960fb4e2ba1d6b03c82f399a9 in term 0.
I20260203 21:06:41.544013 2403 leader_election.cc:304] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e7d723a89424d3484fc057e3ab1a8e0, 730c029960fb4e2ba1d6b03c82f399a9; no voters:
I20260203 21:06:41.544247 2687 raft_consensus.cc:2804] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:06:41.544312 2687 raft_consensus.cc:493] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:06:41.544338 2687 raft_consensus.cc:3060] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:41.545256 2687 raft_consensus.cc:515] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.545599 2687 leader_election.cc:290] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [CANDIDATE]: Term 1 election: Requested vote from peers 0e7d723a89424d3484fc057e3ab1a8e0 (127.0.102.129:37489), 8c3cd322b4244c5ebffbec1606b2d5be (127.0.102.131:39909)
I20260203 21:06:41.545945 2326 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07601e39784e4bf69a9737e75bc03dbf" candidate_uuid: "730c029960fb4e2ba1d6b03c82f399a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e7d723a89424d3484fc057e3ab1a8e0"
I20260203 21:06:41.545945 2612 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "07601e39784e4bf69a9737e75bc03dbf" candidate_uuid: "730c029960fb4e2ba1d6b03c82f399a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3cd322b4244c5ebffbec1606b2d5be"
I20260203 21:06:41.546061 2612 raft_consensus.cc:3060] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:41.546061 2326 raft_consensus.cc:3060] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:41.547066 2326 raft_consensus.cc:2468] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730c029960fb4e2ba1d6b03c82f399a9 in term 1.
I20260203 21:06:41.547066 2612 raft_consensus.cc:2468] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730c029960fb4e2ba1d6b03c82f399a9 in term 1.
I20260203 21:06:41.547353 2401 leader_election.cc:304] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730c029960fb4e2ba1d6b03c82f399a9, 8c3cd322b4244c5ebffbec1606b2d5be; no voters:
I20260203 21:06:41.547547 2687 raft_consensus.cc:2804] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:41.547695 2687 raft_consensus.cc:697] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 LEADER]: Becoming Leader. State: Replica: 730c029960fb4e2ba1d6b03c82f399a9, State: Running, Role: LEADER
I20260203 21:06:41.547978 2687 consensus_queue.cc:237] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.550009 2704 tablet_replica.cc:442] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9: TxnStatusTablet state changed. Reason: New leader 730c029960fb4e2ba1d6b03c82f399a9. Latest consensus state: current_term: 1 leader_uuid: "730c029960fb4e2ba1d6b03c82f399a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } } }
I20260203 21:06:41.550108 2704 tablet_replica.cc:445] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9: This TxnStatusTablet replica's current role is: LEADER
I20260203 21:06:41.550365 2706 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260203 21:06:41.550561 2706 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260203 21:06:41.551110 2169 catalog_manager.cc:5654] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 730c029960fb4e2ba1d6b03c82f399a9 (127.0.102.130). New cstate: current_term: 1 leader_uuid: "730c029960fb4e2ba1d6b03c82f399a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } health_report { overall_health: UNKNOWN } } }
I20260203 21:06:41.579011 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.575563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:46385 (local address 127.0.0.1:47134)
0203 21:06:41.575724 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:41.575740 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:41.575845 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:41.576150 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:41.576152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:41.576159 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0203 21:06:41.576343 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.576348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.577343 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.577346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:41.577773 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.577780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.577872 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.577914 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0203 21:06:41.578758 (+ 844us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0203 21:06:41.578770 (+ 12us) client_negotiation.cc:769] Sending connection context
0203 21:06:41.578835 (+ 65us) client_negotiation.cc:240] Negotiation successful
0203 21:06:41.578901 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260203 21:06:41.579042 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.575683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47134 (local address 127.0.102.130:46385)
0203 21:06:41.575825 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:41.575830 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:41.575846 (+ 16us) server_negotiation.cc:408] Connection header received
0203 21:06:41.575974 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:41.575978 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:41.576046 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:41.576140 (+ 94us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:06:41.576448 (+ 308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.577236 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.577871 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.578056 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.578109 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:06:41.578674 (+ 565us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:06:41.578781 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:41.578856 (+ 75us) server_negotiation.cc:300] Negotiation successful
0203 21:06:41.578912 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260203 21:06:41.584825 2612 raft_consensus.cc:1275] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 FOLLOWER]: Refusing update from remote peer 730c029960fb4e2ba1d6b03c82f399a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:06:41.584936 2326 raft_consensus.cc:1275] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Refusing update from remote peer 730c029960fb4e2ba1d6b03c82f399a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:06:41.585419 2704 consensus_queue.cc:1048] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:06:41.585827 2687 consensus_queue.cc:1048] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:06:41.588802 2688 tablet_replica.cc:442] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be: TxnStatusTablet state changed. Reason: New leader 730c029960fb4e2ba1d6b03c82f399a9. Latest consensus state: current_term: 1 leader_uuid: "730c029960fb4e2ba1d6b03c82f399a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } } }
I20260203 21:06:41.588913 2688 tablet_replica.cc:445] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:06:41.590708 2689 tablet_replica.cc:442] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0: TxnStatusTablet state changed. Reason: New leader 730c029960fb4e2ba1d6b03c82f399a9. Latest consensus state: current_term: 1 leader_uuid: "730c029960fb4e2ba1d6b03c82f399a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } } }
I20260203 21:06:41.590821 2689 tablet_replica.cc:445] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:06:41.593187 2687 tablet_replica.cc:442] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "730c029960fb4e2ba1d6b03c82f399a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } } }
I20260203 21:06:41.593288 2687 tablet_replica.cc:445] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9: This TxnStatusTablet replica's current role is: LEADER
I20260203 21:06:41.593798 2688 tablet_replica.cc:442] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "730c029960fb4e2ba1d6b03c82f399a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } } }
I20260203 21:06:41.593894 2688 tablet_replica.cc:445] T 07601e39784e4bf69a9737e75bc03dbf P 8c3cd322b4244c5ebffbec1606b2d5be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:06:41.594923 2704 tablet_replica.cc:442] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "730c029960fb4e2ba1d6b03c82f399a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } } }
I20260203 21:06:41.595017 2704 tablet_replica.cc:445] T 07601e39784e4bf69a9737e75bc03dbf P 730c029960fb4e2ba1d6b03c82f399a9: This TxnStatusTablet replica's current role is: LEADER
I20260203 21:06:41.596438 2710 mvcc.cc:204] Tried to move back new op lower bound from 7250545875283140608 to 7250545875142860800. Current Snapshot: MvccSnapshot[applied={T|T < 7250545875283140608}]
I20260203 21:06:41.597151 2689 tablet_replica.cc:442] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "730c029960fb4e2ba1d6b03c82f399a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } } }
I20260203 21:06:41.597249 2689 tablet_replica.cc:445] T 07601e39784e4bf69a9737e75bc03dbf P 0e7d723a89424d3484fc057e3ab1a8e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:06:41.597456 2712 mvcc.cc:204] Tried to move back new op lower bound from 7250545875283140608 to 7250545875142860800. Current Snapshot: MvccSnapshot[applied={T|T < 7250545875283140608}]
I20260203 21:06:41.605060 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.601326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44336 (local address 127.0.102.131:39909)
0203 21:06:41.601514 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:41.601518 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:41.601829 (+ 311us) server_negotiation.cc:408] Connection header received
0203 21:06:41.602032 (+ 203us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:41.602036 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:41.602121 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:41.602211 (+ 90us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:06:41.602751 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.603217 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.604086 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.604209 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.604255 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:06:41.604743 (+ 488us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:06:41.604828 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:41.604909 (+ 81us) server_negotiation.cc:300] Negotiation successful
0203 21:06:41.604957 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260203 21:06:41.615345 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.612416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:36567 (local address 127.0.102.130:46385)
0203 21:06:41.612542 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:41.612546 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:41.612668 (+ 122us) server_negotiation.cc:408] Connection header received
0203 21:06:41.612803 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:41.612807 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:41.612865 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:41.612945 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:06:41.613235 (+ 290us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.613984 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.614647 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.614849 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.614894 (+ 45us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:06:41.614989 (+ 95us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:06:41.615059 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:41.615172 (+ 113us) server_negotiation.cc:300] Negotiation successful
0203 21:06:41.615218 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260203 21:06:41.615332 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.612313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:46385 (local address 127.0.102.131:36567)
0203 21:06:41.612578 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:41.612591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:41.612717 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:41.612949 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:41.612951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:41.612957 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0203 21:06:41.613154 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.613158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.614117 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.614121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:41.614545 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.614552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.614648 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.614685 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0203 21:06:41.615089 (+ 404us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0203 21:06:41.615094 (+ 5us) client_negotiation.cc:769] Sending connection context
0203 21:06:41.615152 (+ 58us) client_negotiation.cc:240] Negotiation successful
0203 21:06:41.615210 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
I20260203 21:06:41.622267 2326 raft_consensus.cc:1275] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Refusing update from remote peer 8c3cd322b4244c5ebffbec1606b2d5be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:06:41.622502 2470 raft_consensus.cc:1275] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Refusing update from remote peer 8c3cd322b4244c5ebffbec1606b2d5be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:06:41.623030 2691 consensus_queue.cc:1048] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [LEADER]: Connected to new peer: Peer: permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:06:41.623183 2688 consensus_queue.cc:1048] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:06:41.645723 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.642610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:37443 (local address 127.0.102.131:39909)
0203 21:06:41.642750 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:41.642755 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:41.642872 (+ 117us) server_negotiation.cc:408] Connection header received
0203 21:06:41.643031 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:41.643034 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:41.643083 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:41.643143 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:06:41.643614 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.644221 (+ 607us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.645013 (+ 792us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.645225 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.645276 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:06:41.645360 (+ 84us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:06:41.645431 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:41.645560 (+ 129us) server_negotiation.cc:300] Negotiation successful
0203 21:06:41.645606 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260203 21:06:41.645937 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:41.642472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:39909 (local address 127.0.102.130:37443)
0203 21:06:41.642798 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:41.642810 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:41.642913 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:41.643165 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:41.643168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:41.643176 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0203 21:06:41.643494 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.643503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.644352 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:41.644355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:41.644904 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:41.644913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:41.645014 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:41.645065 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0203 21:06:41.645458 (+ 393us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0203 21:06:41.645465 (+ 7us) client_negotiation.cc:769] Sending connection context
0203 21:06:41.645543 (+ 78us) client_negotiation.cc:240] Negotiation successful
0203 21:06:41.645614 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":117,"threads_started":1}
I20260203 21:06:41.670022 2169 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:06:41.670214 2169 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:06:41.673267 2169 catalog_manager.cc:5941] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5: Sending DeleteTablet for 3 replicas of tablet e3d9f628bd144652a8d235734aa25c1c
I20260203 21:06:41.674141 2592 tablet_service.cc:1552] Processing DeleteTablet for tablet e3d9f628bd144652a8d235734aa25c1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:41 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:44326
I20260203 21:06:41.674429 2308 tablet_service.cc:1552] Processing DeleteTablet for tablet e3d9f628bd144652a8d235734aa25c1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:41 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:36132
I20260203 21:06:41.674947 2733 tablet_replica.cc:333] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0: stopping tablet replica
I20260203 21:06:41.675279 2733 raft_consensus.cc:2243] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:06:41.675580 2169 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260203 21:06:41.675690 2733 raft_consensus.cc:2272] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260203 21:06:41.676064 2169 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:06:41.677439 2733 ts_tablet_manager.cc:1916] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:06:41.679355 2450 tablet_service.cc:1552] Processing DeleteTablet for tablet e3d9f628bd144652a8d235734aa25c1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:41 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:47128
I20260203 21:06:41.680794 2733 ts_tablet_manager.cc:1929] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260203 21:06:41.680935 2733 log.cc:1199] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/wal/wals/e3d9f628bd144652a8d235734aa25c1c
I20260203 21:06:41.681350 2733 ts_tablet_manager.cc:1950] T e3d9f628bd144652a8d235734aa25c1c P 0e7d723a89424d3484fc057e3ab1a8e0: Deleting consensus metadata
I20260203 21:06:41.681591 2592 tablet_service.cc:1505] Processing CreateTablet for tablet 9698aaf6c6a346b3841bf8cb97b66737 (DEFAULT_TABLE table=test-table [id=1cb6c48ee06d4d18afe20987a9b1328a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:41.681881 2592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9698aaf6c6a346b3841bf8cb97b66737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:41.683014 2734 tablet_replica.cc:333] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9: stopping tablet replica
I20260203 21:06:41.683348 2734 raft_consensus.cc:2243] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:06:41.683590 2734 raft_consensus.cc:2272] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:06:41.684108 2682 tablet_bootstrap.cc:492] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be: Bootstrap starting.
I20260203 21:06:41.684967 2734 ts_tablet_manager.cc:1916] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:06:41.685107 2682 tablet_bootstrap.cc:654] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:41.685741 2732 tablet_replica.cc:333] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be: stopping tablet replica
I20260203 21:06:41.685978 2732 raft_consensus.cc:2243] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 LEADER]: Raft consensus shutting down.
I20260203 21:06:41.686273 2450 tablet_service.cc:1505] Processing CreateTablet for tablet 9698aaf6c6a346b3841bf8cb97b66737 (DEFAULT_TABLE table=test-table [id=1cb6c48ee06d4d18afe20987a9b1328a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:41.686537 2450 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9698aaf6c6a346b3841bf8cb97b66737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:41.687793 2308 tablet_service.cc:1505] Processing CreateTablet for tablet 9698aaf6c6a346b3841bf8cb97b66737 (DEFAULT_TABLE table=test-table [id=1cb6c48ee06d4d18afe20987a9b1328a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:41.688012 2732 raft_consensus.cc:2272] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:06:41.688050 2308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9698aaf6c6a346b3841bf8cb97b66737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:41.689076 2734 ts_tablet_manager.cc:1929] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260203 21:06:41.689203 2734 log.cc:1199] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/wal/wals/e3d9f628bd144652a8d235734aa25c1c
I20260203 21:06:41.689280 2732 ts_tablet_manager.cc:1916] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:06:41.689574 2734 ts_tablet_manager.cc:1950] T e3d9f628bd144652a8d235734aa25c1c P 730c029960fb4e2ba1d6b03c82f399a9: Deleting consensus metadata
I20260203 21:06:41.691627 2732 ts_tablet_manager.cc:1929] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260203 21:06:41.691707 2732 log.cc:1199] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/wal/wals/e3d9f628bd144652a8d235734aa25c1c
I20260203 21:06:41.691751 2683 tablet_bootstrap.cc:492] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0: Bootstrap starting.
I20260203 21:06:41.691962 2732 ts_tablet_manager.cc:1950] T e3d9f628bd144652a8d235734aa25c1c P 8c3cd322b4244c5ebffbec1606b2d5be: Deleting consensus metadata
I20260203 21:06:41.691964 2153 catalog_manager.cc:4985] TS 730c029960fb4e2ba1d6b03c82f399a9 (127.0.102.130:46385): tablet e3d9f628bd144652a8d235734aa25c1c (table test-table [id=760e650c04444e41b0f2a04c222b472b]) successfully deleted
I20260203 21:06:41.692740 2683 tablet_bootstrap.cc:654] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:41.693935 2681 tablet_bootstrap.cc:492] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9: Bootstrap starting.
I20260203 21:06:41.694034 2683 tablet_bootstrap.cc:492] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0: No bootstrap required, opened a new log
I20260203 21:06:41.694106 2683 ts_tablet_manager.cc:1403] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260203 21:06:41.694548 2683 raft_consensus.cc:359] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.694650 2683 raft_consensus.cc:385] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:41.694689 2683 raft_consensus.cc:740] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7d723a89424d3484fc057e3ab1a8e0, State: Initialized, Role: FOLLOWER
I20260203 21:06:41.694808 2683 consensus_queue.cc:260] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.694861 2681 tablet_bootstrap.cc:654] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:41.695061 2683 ts_tablet_manager.cc:1434] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260203 21:06:41.695545 2153 catalog_manager.cc:4985] TS 8c3cd322b4244c5ebffbec1606b2d5be (127.0.102.131:39909): tablet e3d9f628bd144652a8d235734aa25c1c (table test-table [id=760e650c04444e41b0f2a04c222b472b]) successfully deleted
I20260203 21:06:41.695963 2155 catalog_manager.cc:4985] TS 0e7d723a89424d3484fc057e3ab1a8e0 (127.0.102.129:37489): tablet e3d9f628bd144652a8d235734aa25c1c (table test-table [id=760e650c04444e41b0f2a04c222b472b]) successfully deleted
I20260203 21:06:41.697261 2681 tablet_bootstrap.cc:492] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9: No bootstrap required, opened a new log
I20260203 21:06:41.697351 2681 ts_tablet_manager.cc:1403] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260203 21:06:41.697804 2681 raft_consensus.cc:359] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.697924 2681 raft_consensus.cc:385] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:41.697960 2681 raft_consensus.cc:740] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730c029960fb4e2ba1d6b03c82f399a9, State: Initialized, Role: FOLLOWER
I20260203 21:06:41.698076 2681 consensus_queue.cc:260] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.698320 2681 ts_tablet_manager.cc:1434] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:06:41.699301 2682 tablet_bootstrap.cc:492] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be: No bootstrap required, opened a new log
I20260203 21:06:41.699401 2682 ts_tablet_manager.cc:1403] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20260203 21:06:41.699847 2682 raft_consensus.cc:359] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.699970 2682 raft_consensus.cc:385] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:41.700001 2682 raft_consensus.cc:740] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3cd322b4244c5ebffbec1606b2d5be, State: Initialized, Role: FOLLOWER
I20260203 21:06:41.700126 2682 consensus_queue.cc:260] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.700364 2682 ts_tablet_manager.cc:1434] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260203 21:06:41.889101 2729 raft_consensus.cc:493] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:06:41.889247 2729 raft_consensus.cc:515] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.889724 2729 leader_election.cc:290] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e7d723a89424d3484fc057e3ab1a8e0 (127.0.102.129:37489), 730c029960fb4e2ba1d6b03c82f399a9 (127.0.102.130:46385)
I20260203 21:06:41.890256 2326 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9698aaf6c6a346b3841bf8cb97b66737" candidate_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" is_pre_election: true
I20260203 21:06:41.890256 2470 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9698aaf6c6a346b3841bf8cb97b66737" candidate_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730c029960fb4e2ba1d6b03c82f399a9" is_pre_election: true
I20260203 21:06:41.890426 2470 raft_consensus.cc:2468] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c3cd322b4244c5ebffbec1606b2d5be in term 0.
I20260203 21:06:41.890452 2326 raft_consensus.cc:2468] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c3cd322b4244c5ebffbec1606b2d5be in term 0.
I20260203 21:06:41.890794 2545 leader_election.cc:304] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e7d723a89424d3484fc057e3ab1a8e0, 730c029960fb4e2ba1d6b03c82f399a9, 8c3cd322b4244c5ebffbec1606b2d5be; no voters:
I20260203 21:06:41.891004 2729 raft_consensus.cc:2804] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:06:41.891067 2729 raft_consensus.cc:493] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:06:41.891098 2729 raft_consensus.cc:3060] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:41.892038 2729 raft_consensus.cc:515] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.892385 2729 leader_election.cc:290] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [CANDIDATE]: Term 1 election: Requested vote from peers 0e7d723a89424d3484fc057e3ab1a8e0 (127.0.102.129:37489), 730c029960fb4e2ba1d6b03c82f399a9 (127.0.102.130:46385)
I20260203 21:06:41.892793 2326 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9698aaf6c6a346b3841bf8cb97b66737" candidate_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e7d723a89424d3484fc057e3ab1a8e0"
I20260203 21:06:41.892800 2470 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9698aaf6c6a346b3841bf8cb97b66737" candidate_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730c029960fb4e2ba1d6b03c82f399a9"
I20260203 21:06:41.892915 2326 raft_consensus.cc:3060] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:41.892921 2470 raft_consensus.cc:3060] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:41.893693 2470 raft_consensus.cc:2468] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3cd322b4244c5ebffbec1606b2d5be in term 1.
I20260203 21:06:41.893814 2326 raft_consensus.cc:2468] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3cd322b4244c5ebffbec1606b2d5be in term 1.
I20260203 21:06:41.894006 2543 leader_election.cc:304] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730c029960fb4e2ba1d6b03c82f399a9, 8c3cd322b4244c5ebffbec1606b2d5be; no voters:
I20260203 21:06:41.894160 2729 raft_consensus.cc:2804] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:41.894241 2729 raft_consensus.cc:697] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 LEADER]: Becoming Leader. State: Replica: 8c3cd322b4244c5ebffbec1606b2d5be, State: Running, Role: LEADER
I20260203 21:06:41.894376 2729 consensus_queue.cc:237] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:41.896034 2167 catalog_manager.cc:5654] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c3cd322b4244c5ebffbec1606b2d5be (127.0.102.131). New cstate: current_term: 1 leader_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } health_report { overall_health: HEALTHY } } }
I20260203 21:06:41.945686 2470 raft_consensus.cc:1275] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Refusing update from remote peer 8c3cd322b4244c5ebffbec1606b2d5be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:06:41.945685 2326 raft_consensus.cc:1275] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Refusing update from remote peer 8c3cd322b4244c5ebffbec1606b2d5be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:06:41.946077 2729 consensus_queue.cc:1048] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:06:41.946278 2691 consensus_queue.cc:1048] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [LEADER]: Connected to new peer: Peer: permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:06:41.949102 2725 mvcc.cc:204] Tried to move back new op lower bound from 7250545876766265344 to 7250545876560424960. Current Snapshot: MvccSnapshot[applied={T|T < 7250545876766265344}]
W20260203 21:06:41.958958 2710 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20260203 21:06:43.508858 2167 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:06:43.509008 2167 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:06:43.511467 2167 catalog_manager.cc:5941] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5: Sending DeleteTablet for 3 replicas of tablet 9698aaf6c6a346b3841bf8cb97b66737
I20260203 21:06:43.512014 2450 tablet_service.cc:1552] Processing DeleteTablet for tablet 9698aaf6c6a346b3841bf8cb97b66737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:43 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:47128
I20260203 21:06:43.512017 2308 tablet_service.cc:1552] Processing DeleteTablet for tablet 9698aaf6c6a346b3841bf8cb97b66737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:43 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:36132
I20260203 21:06:43.512089 2592 tablet_service.cc:1552] Processing DeleteTablet for tablet 9698aaf6c6a346b3841bf8cb97b66737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:43 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:44326
I20260203 21:06:43.512368 2766 tablet_replica.cc:333] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be: stopping tablet replica
I20260203 21:06:43.512360 2764 tablet_replica.cc:333] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9: stopping tablet replica
I20260203 21:06:43.512481 2766 raft_consensus.cc:2243] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 LEADER]: Raft consensus shutting down.
I20260203 21:06:43.512482 2764 raft_consensus.cc:2243] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:06:43.512600 2764 raft_consensus.cc:2272] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:06:43.513062 2766 raft_consensus.cc:2272] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:06:43.513229 2764 ts_tablet_manager.cc:1916] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:06:43.513306 2765 tablet_replica.cc:333] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0: stopping tablet replica
I20260203 21:06:43.513406 2765 raft_consensus.cc:2243] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:06:43.513509 2765 raft_consensus.cc:2272] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:06:43.513674 2766 ts_tablet_manager.cc:1916] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:06:43.514104 2765 ts_tablet_manager.cc:1916] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:06:43.515694 2764 ts_tablet_manager.cc:1929] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260203 21:06:43.515815 2764 log.cc:1199] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/wal/wals/9698aaf6c6a346b3841bf8cb97b66737
I20260203 21:06:43.516160 2764 ts_tablet_manager.cc:1950] T 9698aaf6c6a346b3841bf8cb97b66737 P 730c029960fb4e2ba1d6b03c82f399a9: Deleting consensus metadata
I20260203 21:06:43.516211 2766 ts_tablet_manager.cc:1929] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260203 21:06:43.516337 2766 log.cc:1199] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/wal/wals/9698aaf6c6a346b3841bf8cb97b66737
I20260203 21:06:43.516348 2765 ts_tablet_manager.cc:1929] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260203 21:06:43.516439 2765 log.cc:1199] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/wal/wals/9698aaf6c6a346b3841bf8cb97b66737
I20260203 21:06:43.516714 2766 ts_tablet_manager.cc:1950] T 9698aaf6c6a346b3841bf8cb97b66737 P 8c3cd322b4244c5ebffbec1606b2d5be: Deleting consensus metadata
I20260203 21:06:43.516717 2765 ts_tablet_manager.cc:1950] T 9698aaf6c6a346b3841bf8cb97b66737 P 0e7d723a89424d3484fc057e3ab1a8e0: Deleting consensus metadata
I20260203 21:06:43.516921 2153 catalog_manager.cc:4985] TS 730c029960fb4e2ba1d6b03c82f399a9 (127.0.102.130:46385): tablet 9698aaf6c6a346b3841bf8cb97b66737 (table test-table [id=1cb6c48ee06d4d18afe20987a9b1328a]) successfully deleted
I20260203 21:06:43.517305 2153 catalog_manager.cc:4985] TS 8c3cd322b4244c5ebffbec1606b2d5be (127.0.102.131:39909): tablet 9698aaf6c6a346b3841bf8cb97b66737 (table test-table [id=1cb6c48ee06d4d18afe20987a9b1328a]) successfully deleted
I20260203 21:06:43.519411 2155 catalog_manager.cc:4985] TS 0e7d723a89424d3484fc057e3ab1a8e0 (127.0.102.129:37489): tablet 9698aaf6c6a346b3841bf8cb97b66737 (table test-table [id=1cb6c48ee06d4d18afe20987a9b1328a]) successfully deleted
Feb 03 21:06:43 dist-test-slave-j889 krb5kdc[2121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 03 21:06:43 dist-test-slave-j889 krb5kdc[2121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152803, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:43.536887 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:43.529903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55588 (local address 127.0.102.190:42201)
0203 21:06:43.530176 (+ 273us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:43.530180 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:43.530193 (+ 13us) server_negotiation.cc:408] Connection header received
0203 21:06:43.530233 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:43.530236 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:43.530279 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:43.530353 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:43.531081 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:43.531610 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:43.532226 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:43.532366 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:43.534321 (+ 1955us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:43.534340 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:43.534342 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:43.534369 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:43.535577 (+ 1208us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:43.535914 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:43.535918 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:43.535920 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:43.535968 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:43.536158 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:43.536160 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:43.536162 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:43.536303 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:43.536415 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:43.536561 (+ 146us) server_negotiation.cc:300] Negotiation successful
0203 21:06:43.536701 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":117,"threads_started":1}
I20260203 21:06:43.539140 2167 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260203 21:06:43.539589 2167 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:06:43.544145 2308 tablet_service.cc:1505] Processing CreateTablet for tablet 6ad59999f25f4e56bb4a94fd41da66e9 (DEFAULT_TABLE table=test-table [id=355c59f82ada43218f760dab8b691eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:43.544297 2450 tablet_service.cc:1505] Processing CreateTablet for tablet 6ad59999f25f4e56bb4a94fd41da66e9 (DEFAULT_TABLE table=test-table [id=355c59f82ada43218f760dab8b691eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:43.544443 2308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ad59999f25f4e56bb4a94fd41da66e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:43.544552 2450 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ad59999f25f4e56bb4a94fd41da66e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:43.546327 2775 tablet_bootstrap.cc:492] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0: Bootstrap starting.
I20260203 21:06:43.546373 2776 tablet_bootstrap.cc:492] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9: Bootstrap starting.
I20260203 21:06:43.546806 2592 tablet_service.cc:1505] Processing CreateTablet for tablet 6ad59999f25f4e56bb4a94fd41da66e9 (DEFAULT_TABLE table=test-table [id=355c59f82ada43218f760dab8b691eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:43.547081 2592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ad59999f25f4e56bb4a94fd41da66e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:43.547410 2775 tablet_bootstrap.cc:654] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:43.547407 2776 tablet_bootstrap.cc:654] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:43.548429 2775 tablet_bootstrap.cc:492] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0: No bootstrap required, opened a new log
I20260203 21:06:43.548521 2775 ts_tablet_manager.cc:1403] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260203 21:06:43.548703 2776 tablet_bootstrap.cc:492] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9: No bootstrap required, opened a new log
I20260203 21:06:43.548784 2776 ts_tablet_manager.cc:1403] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260203 21:06:43.549041 2775 raft_consensus.cc:359] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.549142 2775 raft_consensus.cc:385] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:43.549172 2775 raft_consensus.cc:740] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7d723a89424d3484fc057e3ab1a8e0, State: Initialized, Role: FOLLOWER
I20260203 21:06:43.549201 2776 raft_consensus.cc:359] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.549270 2779 tablet_bootstrap.cc:492] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be: Bootstrap starting.
I20260203 21:06:43.549302 2776 raft_consensus.cc:385] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:43.549330 2776 raft_consensus.cc:740] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730c029960fb4e2ba1d6b03c82f399a9, State: Initialized, Role: FOLLOWER
I20260203 21:06:43.549324 2775 consensus_queue.cc:260] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.549449 2776 consensus_queue.cc:260] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.549685 2775 ts_tablet_manager.cc:1434] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260203 21:06:43.549687 2776 ts_tablet_manager.cc:1434] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:06:43.550129 2779 tablet_bootstrap.cc:654] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:43.551537 2779 tablet_bootstrap.cc:492] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be: No bootstrap required, opened a new log
I20260203 21:06:43.551631 2779 ts_tablet_manager.cc:1403] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260203 21:06:43.551988 2779 raft_consensus.cc:359] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.552078 2779 raft_consensus.cc:385] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:43.552106 2779 raft_consensus.cc:740] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3cd322b4244c5ebffbec1606b2d5be, State: Initialized, Role: FOLLOWER
I20260203 21:06:43.552237 2779 consensus_queue.cc:260] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.552526 2779 ts_tablet_manager.cc:1434] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:06:43.636837 2722 raft_consensus.cc:493] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:06:43.636976 2722 raft_consensus.cc:515] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.637415 2722 leader_election.cc:290] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e7d723a89424d3484fc057e3ab1a8e0 (127.0.102.129:37489), 8c3cd322b4244c5ebffbec1606b2d5be (127.0.102.131:39909)
I20260203 21:06:43.637884 2612 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ad59999f25f4e56bb4a94fd41da66e9" candidate_uuid: "730c029960fb4e2ba1d6b03c82f399a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" is_pre_election: true
I20260203 21:06:43.638062 2612 raft_consensus.cc:2468] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730c029960fb4e2ba1d6b03c82f399a9 in term 0.
I20260203 21:06:43.638092 2328 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ad59999f25f4e56bb4a94fd41da66e9" candidate_uuid: "730c029960fb4e2ba1d6b03c82f399a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" is_pre_election: true
I20260203 21:06:43.638219 2328 raft_consensus.cc:2468] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730c029960fb4e2ba1d6b03c82f399a9 in term 0.
I20260203 21:06:43.638397 2401 leader_election.cc:304] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730c029960fb4e2ba1d6b03c82f399a9, 8c3cd322b4244c5ebffbec1606b2d5be; no voters:
I20260203 21:06:43.638554 2722 raft_consensus.cc:2804] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:06:43.638604 2722 raft_consensus.cc:493] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:06:43.638629 2722 raft_consensus.cc:3060] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:43.639508 2722 raft_consensus.cc:515] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.639832 2722 leader_election.cc:290] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [CANDIDATE]: Term 1 election: Requested vote from peers 0e7d723a89424d3484fc057e3ab1a8e0 (127.0.102.129:37489), 8c3cd322b4244c5ebffbec1606b2d5be (127.0.102.131:39909)
I20260203 21:06:43.640156 2328 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ad59999f25f4e56bb4a94fd41da66e9" candidate_uuid: "730c029960fb4e2ba1d6b03c82f399a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e7d723a89424d3484fc057e3ab1a8e0"
I20260203 21:06:43.640177 2612 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ad59999f25f4e56bb4a94fd41da66e9" candidate_uuid: "730c029960fb4e2ba1d6b03c82f399a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3cd322b4244c5ebffbec1606b2d5be"
I20260203 21:06:43.640265 2328 raft_consensus.cc:3060] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:43.640288 2612 raft_consensus.cc:3060] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:43.641139 2612 raft_consensus.cc:2468] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730c029960fb4e2ba1d6b03c82f399a9 in term 1.
I20260203 21:06:43.641188 2328 raft_consensus.cc:2468] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730c029960fb4e2ba1d6b03c82f399a9 in term 1.
I20260203 21:06:43.641444 2401 leader_election.cc:304] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 730c029960fb4e2ba1d6b03c82f399a9, 8c3cd322b4244c5ebffbec1606b2d5be; no voters:
I20260203 21:06:43.641672 2722 raft_consensus.cc:2804] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:43.641757 2722 raft_consensus.cc:697] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 LEADER]: Becoming Leader. State: Replica: 730c029960fb4e2ba1d6b03c82f399a9, State: Running, Role: LEADER
I20260203 21:06:43.641898 2722 consensus_queue.cc:237] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.643575 2168 catalog_manager.cc:5654] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 730c029960fb4e2ba1d6b03c82f399a9 (127.0.102.130). New cstate: current_term: 1 leader_uuid: "730c029960fb4e2ba1d6b03c82f399a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } health_report { overall_health: UNKNOWN } } }
I20260203 21:06:43.678321 2784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:43.675591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47148 (local address 127.0.102.130:46385)
0203 21:06:43.675844 (+ 253us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:43.675848 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:43.675861 (+ 13us) server_negotiation.cc:408] Connection header received
0203 21:06:43.675903 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:43.675919 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:43.675964 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:43.676033 (+ 69us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:06:43.676367 (+ 334us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:43.676897 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:43.677646 (+ 749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:43.677834 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:43.677877 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:06:43.677954 (+ 77us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:06:43.678014 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:43.678125 (+ 111us) server_negotiation.cc:300] Negotiation successful
0203 21:06:43.678175 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
I20260203 21:06:43.686054 2784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:43.683801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:37325 (local address 127.0.102.130:46385)
0203 21:06:43.683909 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:43.683912 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:43.684011 (+ 99us) server_negotiation.cc:408] Connection header received
0203 21:06:43.684099 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:43.684101 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:43.684144 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:43.684203 (+ 59us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:06:43.684492 (+ 289us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:43.684978 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:43.685508 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:43.685636 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:43.685692 (+ 56us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:06:43.685755 (+ 63us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:06:43.685812 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:43.685895 (+ 83us) server_negotiation.cc:300] Negotiation successful
0203 21:06:43.685931 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260203 21:06:43.686060 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:43.683713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:46385 (local address 127.0.102.130:37325)
0203 21:06:43.683938 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:43.683951 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:43.684033 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:43.684215 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:43.684217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:43.684223 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0203 21:06:43.684416 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:43.684421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:43.685061 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:43.685064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:43.685434 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:43.685439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:43.685518 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:43.685538 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0203 21:06:43.685827 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0203 21:06:43.685833 (+ 6us) client_negotiation.cc:769] Sending connection context
0203 21:06:43.685882 (+ 49us) client_negotiation.cc:240] Negotiation successful
0203 21:06:43.685936 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260203 21:06:43.691326 2612 raft_consensus.cc:1275] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 FOLLOWER]: Refusing update from remote peer 730c029960fb4e2ba1d6b03c82f399a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:06:43.691326 2328 raft_consensus.cc:1275] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Refusing update from remote peer 730c029960fb4e2ba1d6b03c82f399a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:06:43.691673 2745 consensus_queue.cc:1048] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:06:43.691795 2750 consensus_queue.cc:1048] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:06:43.724884 2168 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:06:43.725036 2168 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:06:43.727059 2168 catalog_manager.cc:5941] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5: Sending DeleteTablet for 3 replicas of tablet 6ad59999f25f4e56bb4a94fd41da66e9
I20260203 21:06:43.727784 2592 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ad59999f25f4e56bb4a94fd41da66e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:43 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:44326
I20260203 21:06:43.727788 2308 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ad59999f25f4e56bb4a94fd41da66e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:43 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:36132
I20260203 21:06:43.727982 2765 tablet_replica.cc:333] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0: stopping tablet replica
I20260203 21:06:43.727983 2766 tablet_replica.cc:333] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be: stopping tablet replica
I20260203 21:06:43.728084 2766 raft_consensus.cc:2243] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:06:43.728096 2765 raft_consensus.cc:2243] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:06:43.728186 2766 raft_consensus.cc:2272] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:06:43.728200 2765 raft_consensus.cc:2272] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:06:43.728706 2168 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260203 21:06:43.728972 2766 ts_tablet_manager.cc:1916] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260203 21:06:43.729136 2168 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:06:43.729221 2765 ts_tablet_manager.cc:1916] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:06:43.731498 2765 ts_tablet_manager.cc:1929] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260203 21:06:43.731509 2766 ts_tablet_manager.cc:1929] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260203 21:06:43.731534 2450 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ad59999f25f4e56bb4a94fd41da66e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:43 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:47128
I20260203 21:06:43.731573 2765 log.cc:1199] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-0/wal/wals/6ad59999f25f4e56bb4a94fd41da66e9
I20260203 21:06:43.731580 2766 log.cc:1199] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/wal/wals/6ad59999f25f4e56bb4a94fd41da66e9
I20260203 21:06:43.731711 2764 tablet_replica.cc:333] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9: stopping tablet replica
I20260203 21:06:43.731822 2765 ts_tablet_manager.cc:1950] T 6ad59999f25f4e56bb4a94fd41da66e9 P 0e7d723a89424d3484fc057e3ab1a8e0: Deleting consensus metadata
I20260203 21:06:43.731825 2766 ts_tablet_manager.cc:1950] T 6ad59999f25f4e56bb4a94fd41da66e9 P 8c3cd322b4244c5ebffbec1606b2d5be: Deleting consensus metadata
I20260203 21:06:43.731832 2764 raft_consensus.cc:2243] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 LEADER]: Raft consensus shutting down.
I20260203 21:06:43.732169 2764 raft_consensus.cc:2272] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:06:43.732676 2155 catalog_manager.cc:4985] TS 0e7d723a89424d3484fc057e3ab1a8e0 (127.0.102.129:37489): tablet 6ad59999f25f4e56bb4a94fd41da66e9 (table test-table [id=355c59f82ada43218f760dab8b691eae]) successfully deleted
I20260203 21:06:43.732686 2153 catalog_manager.cc:4985] TS 8c3cd322b4244c5ebffbec1606b2d5be (127.0.102.131:39909): tablet 6ad59999f25f4e56bb4a94fd41da66e9 (table test-table [id=355c59f82ada43218f760dab8b691eae]) successfully deleted
I20260203 21:06:43.732903 2764 ts_tablet_manager.cc:1916] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:06:43.734699 2764 ts_tablet_manager.cc:1929] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260203 21:06:43.734771 2764 log.cc:1199] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/wal/wals/6ad59999f25f4e56bb4a94fd41da66e9
I20260203 21:06:43.734954 2764 ts_tablet_manager.cc:1950] T 6ad59999f25f4e56bb4a94fd41da66e9 P 730c029960fb4e2ba1d6b03c82f399a9: Deleting consensus metadata
I20260203 21:06:43.735625 2153 catalog_manager.cc:4985] TS 730c029960fb4e2ba1d6b03c82f399a9 (127.0.102.130:46385): tablet 6ad59999f25f4e56bb4a94fd41da66e9 (table test-table [id=355c59f82ada43218f760dab8b691eae]) successfully deleted
I20260203 21:06:43.737805 2308 tablet_service.cc:1505] Processing CreateTablet for tablet 1aff92538c9b4f4c90216ae73bc236a6 (DEFAULT_TABLE table=test-table [id=ece14178aa064731bd93c6d35ff70ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:43.738081 2308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aff92538c9b4f4c90216ae73bc236a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:43.738142 2592 tablet_service.cc:1505] Processing CreateTablet for tablet 1aff92538c9b4f4c90216ae73bc236a6 (DEFAULT_TABLE table=test-table [id=ece14178aa064731bd93c6d35ff70ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:43.738358 2592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aff92538c9b4f4c90216ae73bc236a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:43.739998 2779 tablet_bootstrap.cc:492] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be: Bootstrap starting.
I20260203 21:06:43.740163 2775 tablet_bootstrap.cc:492] T 1aff92538c9b4f4c90216ae73bc236a6 P 0e7d723a89424d3484fc057e3ab1a8e0: Bootstrap starting.
I20260203 21:06:43.740362 2450 tablet_service.cc:1505] Processing CreateTablet for tablet 1aff92538c9b4f4c90216ae73bc236a6 (DEFAULT_TABLE table=test-table [id=ece14178aa064731bd93c6d35ff70ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:06:43.740597 2450 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aff92538c9b4f4c90216ae73bc236a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:43.740907 2779 tablet_bootstrap.cc:654] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:43.741169 2775 tablet_bootstrap.cc:654] T 1aff92538c9b4f4c90216ae73bc236a6 P 0e7d723a89424d3484fc057e3ab1a8e0: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:43.742391 2775 tablet_bootstrap.cc:492] T 1aff92538c9b4f4c90216ae73bc236a6 P 0e7d723a89424d3484fc057e3ab1a8e0: No bootstrap required, opened a new log
I20260203 21:06:43.742470 2775 ts_tablet_manager.cc:1403] T 1aff92538c9b4f4c90216ae73bc236a6 P 0e7d723a89424d3484fc057e3ab1a8e0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260203 21:06:43.742898 2775 raft_consensus.cc:359] T 1aff92538c9b4f4c90216ae73bc236a6 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.742996 2775 raft_consensus.cc:385] T 1aff92538c9b4f4c90216ae73bc236a6 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:43.743000 2779 tablet_bootstrap.cc:492] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be: No bootstrap required, opened a new log
I20260203 21:06:43.743034 2775 raft_consensus.cc:740] T 1aff92538c9b4f4c90216ae73bc236a6 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e7d723a89424d3484fc057e3ab1a8e0, State: Initialized, Role: FOLLOWER
I20260203 21:06:43.743068 2779 ts_tablet_manager.cc:1403] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260203 21:06:43.743149 2775 consensus_queue.cc:260] T 1aff92538c9b4f4c90216ae73bc236a6 P 0e7d723a89424d3484fc057e3ab1a8e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.743392 2775 ts_tablet_manager.cc:1434] T 1aff92538c9b4f4c90216ae73bc236a6 P 0e7d723a89424d3484fc057e3ab1a8e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260203 21:06:43.743427 2779 raft_consensus.cc:359] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.743503 2779 raft_consensus.cc:385] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:43.743523 2779 raft_consensus.cc:740] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3cd322b4244c5ebffbec1606b2d5be, State: Initialized, Role: FOLLOWER
I20260203 21:06:43.743590 2776 tablet_bootstrap.cc:492] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9: Bootstrap starting.
I20260203 21:06:43.743618 2779 consensus_queue.cc:260] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.743842 2779 ts_tablet_manager.cc:1434] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:06:43.744395 2776 tablet_bootstrap.cc:654] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:43.745477 2776 tablet_bootstrap.cc:492] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9: No bootstrap required, opened a new log
I20260203 21:06:43.745556 2776 ts_tablet_manager.cc:1403] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260203 21:06:43.745922 2776 raft_consensus.cc:359] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.746011 2776 raft_consensus.cc:385] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:43.746052 2776 raft_consensus.cc:740] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730c029960fb4e2ba1d6b03c82f399a9, State: Initialized, Role: FOLLOWER
I20260203 21:06:43.746157 2776 consensus_queue.cc:260] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.746388 2776 ts_tablet_manager.cc:1434] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260203 21:06:43.869269 2755 raft_consensus.cc:493] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:06:43.869402 2755 raft_consensus.cc:515] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.869886 2755 leader_election.cc:290] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e7d723a89424d3484fc057e3ab1a8e0 (127.0.102.129:37489), 730c029960fb4e2ba1d6b03c82f399a9 (127.0.102.130:46385)
I20260203 21:06:43.870311 2326 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aff92538c9b4f4c90216ae73bc236a6" candidate_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" is_pre_election: true
I20260203 21:06:43.870378 2470 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aff92538c9b4f4c90216ae73bc236a6" candidate_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730c029960fb4e2ba1d6b03c82f399a9" is_pre_election: true
I20260203 21:06:43.870460 2326 raft_consensus.cc:2468] T 1aff92538c9b4f4c90216ae73bc236a6 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c3cd322b4244c5ebffbec1606b2d5be in term 0.
I20260203 21:06:43.870563 2470 raft_consensus.cc:2468] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c3cd322b4244c5ebffbec1606b2d5be in term 0.
I20260203 21:06:43.870774 2545 leader_election.cc:304] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e7d723a89424d3484fc057e3ab1a8e0, 8c3cd322b4244c5ebffbec1606b2d5be; no voters:
I20260203 21:06:43.870998 2755 raft_consensus.cc:2804] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:06:43.871054 2755 raft_consensus.cc:493] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:06:43.871086 2755 raft_consensus.cc:3060] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:43.872092 2755 raft_consensus.cc:515] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.872465 2755 leader_election.cc:290] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [CANDIDATE]: Term 1 election: Requested vote from peers 0e7d723a89424d3484fc057e3ab1a8e0 (127.0.102.129:37489), 730c029960fb4e2ba1d6b03c82f399a9 (127.0.102.130:46385)
I20260203 21:06:43.872855 2326 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aff92538c9b4f4c90216ae73bc236a6" candidate_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e7d723a89424d3484fc057e3ab1a8e0"
I20260203 21:06:43.872867 2470 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1aff92538c9b4f4c90216ae73bc236a6" candidate_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730c029960fb4e2ba1d6b03c82f399a9"
I20260203 21:06:43.872980 2326 raft_consensus.cc:3060] T 1aff92538c9b4f4c90216ae73bc236a6 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:43.872987 2470 raft_consensus.cc:3060] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:43.873822 2326 raft_consensus.cc:2468] T 1aff92538c9b4f4c90216ae73bc236a6 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3cd322b4244c5ebffbec1606b2d5be in term 1.
I20260203 21:06:43.874063 2470 raft_consensus.cc:2468] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3cd322b4244c5ebffbec1606b2d5be in term 1.
I20260203 21:06:43.874125 2545 leader_election.cc:304] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e7d723a89424d3484fc057e3ab1a8e0, 8c3cd322b4244c5ebffbec1606b2d5be; no voters:
I20260203 21:06:43.874313 2755 raft_consensus.cc:2804] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:43.874404 2755 raft_consensus.cc:697] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 LEADER]: Becoming Leader. State: Replica: 8c3cd322b4244c5ebffbec1606b2d5be, State: Running, Role: LEADER
I20260203 21:06:43.874534 2755 consensus_queue.cc:237] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } }
I20260203 21:06:43.876385 2167 catalog_manager.cc:5654] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c3cd322b4244c5ebffbec1606b2d5be (127.0.102.131). New cstate: current_term: 1 leader_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3cd322b4244c5ebffbec1606b2d5be" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 39909 } health_report { overall_health: HEALTHY } } }
I20260203 21:06:43.909330 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:43.906971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44350 (local address 127.0.102.131:39909)
0203 21:06:43.907250 (+ 279us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:43.907253 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:43.907264 (+ 11us) server_negotiation.cc:408] Connection header received
0203 21:06:43.907300 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:43.907302 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:43.907344 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:43.907412 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:06:43.907692 (+ 280us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:43.908215 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:43.908771 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:43.908920 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:43.908957 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:06:43.909017 (+ 60us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:06:43.909073 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:43.909144 (+ 71us) server_negotiation.cc:300] Negotiation successful
0203 21:06:43.909186 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":125,"threads_started":1}
I20260203 21:06:43.915688 2326 raft_consensus.cc:1275] T 1aff92538c9b4f4c90216ae73bc236a6 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Refusing update from remote peer 8c3cd322b4244c5ebffbec1606b2d5be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:06:43.915688 2470 raft_consensus.cc:1275] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Refusing update from remote peer 8c3cd322b4244c5ebffbec1606b2d5be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:06:43.916133 2755 consensus_queue.cc:1048] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e7d723a89424d3484fc057e3ab1a8e0" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:06:43.916306 2758 consensus_queue.cc:1048] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [LEADER]: Connected to new peer: Peer: permanent_uuid: "730c029960fb4e2ba1d6b03c82f399a9" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 46385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260203 21:06:43.929828 2747 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20260203 21:06:45.478039 2167 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:06:45.478202 2167 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:06:45.480558 2167 catalog_manager.cc:5941] T 00000000000000000000000000000000 P bf2ff9d5092a44e8871418ca068b54b5: Sending DeleteTablet for 3 replicas of tablet 1aff92538c9b4f4c90216ae73bc236a6
I20260203 21:06:45.481137 2308 tablet_service.cc:1552] Processing DeleteTablet for tablet 1aff92538c9b4f4c90216ae73bc236a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:45 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:36132
I20260203 21:06:45.481242 2592 tablet_service.cc:1552] Processing DeleteTablet for tablet 1aff92538c9b4f4c90216ae73bc236a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:45 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:44326
I20260203 21:06:45.481477 2450 tablet_service.cc:1552] Processing DeleteTablet for tablet 1aff92538c9b4f4c90216ae73bc236a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:06:45 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:47128
I20260203 21:06:45.481511 2822 tablet_replica.cc:333] T 1aff92538c9b4f4c90216ae73bc236a6 P 0e7d723a89424d3484fc057e3ab1a8e0: stopping tablet replica
I20260203 21:06:45.481626 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 2243
I20260203 21:06:45.481634 2822 raft_consensus.cc:2243] T 1aff92538c9b4f4c90216ae73bc236a6 P 0e7d723a89424d3484fc057e3ab1a8e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:06:45.481537 2823 tablet_replica.cc:333] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be: stopping tablet replica
I20260203 21:06:45.482630 2824 tablet_replica.cc:333] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9: stopping tablet replica
I20260203 21:06:45.482800 2824 raft_consensus.cc:2243] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:06:45.482932 2824 raft_consensus.cc:2272] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:06:45.483364 2823 raft_consensus.cc:2243] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 LEADER]: Raft consensus shutting down.
I20260203 21:06:45.483575 2823 raft_consensus.cc:2272] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:06:45.484076 2823 ts_tablet_manager.cc:1916] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:06:45.484076 2824 ts_tablet_manager.cc:1916] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:06:45.486510 2824 ts_tablet_manager.cc:1929] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260203 21:06:45.486510 2823 ts_tablet_manager.cc:1929] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260203 21:06:45.486594 2823 log.cc:1199] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-2/wal/wals/1aff92538c9b4f4c90216ae73bc236a6
I20260203 21:06:45.486594 2824 log.cc:1199] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1770152788143875-410-0/minicluster-data/ts-1/wal/wals/1aff92538c9b4f4c90216ae73bc236a6
I20260203 21:06:45.486888 2823 ts_tablet_manager.cc:1950] T 1aff92538c9b4f4c90216ae73bc236a6 P 8c3cd322b4244c5ebffbec1606b2d5be: Deleting consensus metadata
I20260203 21:06:45.486888 2824 ts_tablet_manager.cc:1950] T 1aff92538c9b4f4c90216ae73bc236a6 P 730c029960fb4e2ba1d6b03c82f399a9: Deleting consensus metadata
W20260203 21:06:45.487373 2155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.102.129:37489 (error 108)
I20260203 21:06:45.487597 2153 catalog_manager.cc:4985] TS 730c029960fb4e2ba1d6b03c82f399a9 (127.0.102.130:46385): tablet 1aff92538c9b4f4c90216ae73bc236a6 (table test-table [id=ece14178aa064731bd93c6d35ff70ab4]) successfully deleted
I20260203 21:06:45.487768 2153 catalog_manager.cc:4985] TS 8c3cd322b4244c5ebffbec1606b2d5be (127.0.102.131:39909): tablet 1aff92538c9b4f4c90216ae73bc236a6 (table test-table [id=ece14178aa064731bd93c6d35ff70ab4]) successfully deleted
I20260203 21:06:45.488149 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 2385
I20260203 21:06:45.489028 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:45.488064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:37489 (local address 127.0.0.1:36138)
0203 21:06:45.488866 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:45.488925 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.129:37489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":89,"threads_started":1}
W20260203 21:06:45.489159 2155 catalog_manager.cc:4712] TS 0e7d723a89424d3484fc057e3ab1a8e0 (127.0.102.129:37489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1aff92538c9b4f4c90216ae73bc236a6: Network error: Client connection negotiation failed: client connection to 127.0.102.129:37489: connect: Connection refused (error 111)
I20260203 21:06:45.495245 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 2527
I20260203 21:06:45.501036 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 2137
2026-02-03T21:06:45Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7364 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:06:45 dist-test-slave-j889 krb5kdc[2829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:06:45 dist-test-slave-j889 krb5kdc[2829](info): set up 2 sockets
Feb 03 21:06:45 dist-test-slave-j889 krb5kdc[2829](info): commencing operation
krb5kdc: starting...
W20260203 21:06:47.344584 410 mini_kdc.cc:121] Time spent starting KDC: real 1.822s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:06:47 dist-test-slave-j889 krb5kdc[2829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:06:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:06:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:47.473829 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:40141
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:43053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:40141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:47.577395 2845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:47.577618 2845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:47.577675 2845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:47.580870 2845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:06:47.580932 2845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:47.580957 2845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:47.580984 2845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:06:47.581004 2845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:06:47.585107 2845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:43053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:40141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:40141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:47.586112 2845 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:47.586926 2845 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:47.592593 2851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:47.592592 2853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:47.592669 2850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:47.592837 2845 server_base.cc:1060] running on GCE node
I20260203 21:06:47.593439 2845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:47.594300 2845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:47.595449 2845 hybrid_clock.cc:648] HybridClock initialized: now 1770152807595426 us; error 39 us; skew 500 ppm
Feb 03 21:06:47 dist-test-slave-j889 krb5kdc[2829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:47.597823 2845 init.cc:377] Logged in from keytab as kudu/127.0.102.190@KRBTEST.COM (short username kudu)
I20260203 21:06:47.598851 2845 webserver.cc:492] Webserver started at http://127.0.102.190:35783/ using document root <none> and password file <none>
I20260203 21:06:47.599378 2845 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:47.599431 2845 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:47.599601 2845 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:47.601328 2845 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "898702932054424497d95541cbf19f2d"
format_stamp: "Formatted at 2026-02-03 21:06:47 on dist-test-slave-j889"
server_key: "6e0fb164d339e6a8928443f681d65069"
server_key_iv: "51e0b5788ef4e3c440a88b6d716044bf"
server_key_version: "encryptionkey@0"
I20260203 21:06:47.601796 2845 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "898702932054424497d95541cbf19f2d"
format_stamp: "Formatted at 2026-02-03 21:06:47 on dist-test-slave-j889"
server_key: "6e0fb164d339e6a8928443f681d65069"
server_key_iv: "51e0b5788ef4e3c440a88b6d716044bf"
server_key_version: "encryptionkey@0"
I20260203 21:06:47.605271 2845 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260203 21:06:47.607621 2860 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:47.608764 2845 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:06:47.608879 2845 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "898702932054424497d95541cbf19f2d"
format_stamp: "Formatted at 2026-02-03 21:06:47 on dist-test-slave-j889"
server_key: "6e0fb164d339e6a8928443f681d65069"
server_key_iv: "51e0b5788ef4e3c440a88b6d716044bf"
server_key_version: "encryptionkey@0"
I20260203 21:06:47.608973 2845 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:47.632769 2845 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:47.635527 2845 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:47.635702 2845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:47.642684 2912 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:40141 every 8 connection(s)
I20260203 21:06:47.642690 2845 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:40141
I20260203 21:06:47.643726 2845 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:06:47.646543 2913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:47.649225 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 2845
I20260203 21:06:47.649351 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:06:47.649595 410 external_mini_cluster.cc:1468] Setting key 44259b4ef913cc82b8ae69dcabfc7a43
I20260203 21:06:47.652451 2913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d: Bootstrap starting.
I20260203 21:06:47.654556 2913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:47.655301 2913 log.cc:826] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d: Log is configured to *not* fsync() on all Append() calls
Feb 03 21:06:47 dist-test-slave-j889 krb5kdc[2829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:47.657061 2913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d: No bootstrap required, opened a new log
I20260203 21:06:47.659588 2913 raft_consensus.cc:359] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898702932054424497d95541cbf19f2d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 40141 } }
I20260203 21:06:47.659766 2913 raft_consensus.cc:385] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:47.659839 2913 raft_consensus.cc:740] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 898702932054424497d95541cbf19f2d, State: Initialized, Role: FOLLOWER
I20260203 21:06:47.660319 2913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898702932054424497d95541cbf19f2d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 40141 } }
I20260203 21:06:47.660475 2913 raft_consensus.cc:399] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:06:47.660545 2913 raft_consensus.cc:493] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:06:47.660651 2913 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:47.661124 2916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:47.650784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49392 (local address 127.0.102.190:40141)
0203 21:06:47.651187 (+ 403us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:47.651197 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:47.651231 (+ 34us) server_negotiation.cc:408] Connection header received
0203 21:06:47.651749 (+ 518us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:47.651767 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:47.652050 (+ 283us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:47.652343 (+ 293us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:47.653272 (+ 929us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:47.654017 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:47.654673 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:47.654932 (+ 259us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:47.656928 (+ 1996us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:47.656946 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:47.656956 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:47.656988 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:47.658633 (+ 1645us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:47.659042 (+ 409us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:47.659048 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:47.659052 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:47.659124 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:47.659364 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:47.659368 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:47.659370 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:47.659701 (+ 331us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:47.659893 (+ 192us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:47.660169 (+ 276us) server_negotiation.cc:300] Negotiation successful
0203 21:06:47.660406 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":129,"threads_started":1}
I20260203 21:06:47.661473 2913 raft_consensus.cc:515] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898702932054424497d95541cbf19f2d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 40141 } }
I20260203 21:06:47.661814 2913 leader_election.cc:304] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 898702932054424497d95541cbf19f2d; no voters:
I20260203 21:06:47.662074 2913 leader_election.cc:290] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:06:47.662143 2918 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:47.662331 2918 raft_consensus.cc:697] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [term 1 LEADER]: Becoming Leader. State: Replica: 898702932054424497d95541cbf19f2d, State: Running, Role: LEADER
I20260203 21:06:47.662662 2918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898702932054424497d95541cbf19f2d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 40141 } }
I20260203 21:06:47.663188 2913 sys_catalog.cc:565] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:06:47.664125 2919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "898702932054424497d95541cbf19f2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898702932054424497d95541cbf19f2d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 40141 } } }
I20260203 21:06:47.664228 2919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [sys.catalog]: This master's current role is: LEADER
I20260203 21:06:47.664584 2927 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:06:47.664729 2920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 898702932054424497d95541cbf19f2d. Latest consensus state: current_term: 1 leader_uuid: "898702932054424497d95541cbf19f2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898702932054424497d95541cbf19f2d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 40141 } } }
I20260203 21:06:47.664873 2920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d [sys.catalog]: This master's current role is: LEADER
I20260203 21:06:47.667573 2927 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:06:47.672232 2927 catalog_manager.cc:1357] Generated new cluster ID: d2092bba298f41e4a28d12c24330e7d9
I20260203 21:06:47.672297 2927 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:06:47.681527 2927 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:06:47.682703 2927 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:06:47.699223 2927 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 898702932054424497d95541cbf19f2d: Generated new TSK 0
I20260203 21:06:47.699960 2927 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:47.749001 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:40141
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:43053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:47.858049 2941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:47.858264 2941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:47.858350 2941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:47.861586 2941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:47.861642 2941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:47.861737 2941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:06:47.865907 2941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:43053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:40141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:47.866959 2941 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:47.867805 2941 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:47.874034 2947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:47.874037 2949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:47.874181 2941 server_base.cc:1060] running on GCE node
W20260203 21:06:47.874038 2946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:47.874552 2941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:47.875036 2941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:47.876187 2941 hybrid_clock.cc:648] HybridClock initialized: now 1770152807876173 us; error 33 us; skew 500 ppm
Feb 03 21:06:47 dist-test-slave-j889 krb5kdc[2829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:47.878608 2941 init.cc:377] Logged in from keytab as kudu/127.0.102.129@KRBTEST.COM (short username kudu)
I20260203 21:06:47.879526 2941 webserver.cc:492] Webserver started at http://127.0.102.129:40235/ using document root <none> and password file <none>
I20260203 21:06:47.880029 2941 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:47.880076 2941 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:47.880242 2941 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:47.881870 2941 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "4737be95d6dd452596f0e6bb7a87eb78"
format_stamp: "Formatted at 2026-02-03 21:06:47 on dist-test-slave-j889"
server_key: "749a3043087499224170827a8ce1a070"
server_key_iv: "83e35d8744a9c0ffb4e411a6dc2fc31e"
server_key_version: "encryptionkey@0"
I20260203 21:06:47.882304 2941 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "4737be95d6dd452596f0e6bb7a87eb78"
format_stamp: "Formatted at 2026-02-03 21:06:47 on dist-test-slave-j889"
server_key: "749a3043087499224170827a8ce1a070"
server_key_iv: "83e35d8744a9c0ffb4e411a6dc2fc31e"
server_key_version: "encryptionkey@0"
I20260203 21:06:47.885386 2941 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260203 21:06:47.887607 2956 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:47.888586 2941 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:06:47.888712 2941 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "4737be95d6dd452596f0e6bb7a87eb78"
format_stamp: "Formatted at 2026-02-03 21:06:47 on dist-test-slave-j889"
server_key: "749a3043087499224170827a8ce1a070"
server_key_iv: "83e35d8744a9c0ffb4e411a6dc2fc31e"
server_key_version: "encryptionkey@0"
I20260203 21:06:47.888803 2941 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:47.910921 2941 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:47.913683 2941 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:47.913844 2941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:47.914359 2941 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:47.915189 2941 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:47.915234 2941 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:47.915285 2941 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:47.915313 2941 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:47.923401 2941 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:46733
I20260203 21:06:47.923429 3069 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:46733 every 8 connection(s)
I20260203 21:06:47.924278 2941 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
Feb 03 21:06:47 dist-test-slave-j889 krb5kdc[2829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:47.934521 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 2941
I20260203 21:06:47.934648 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:06:47.934875 410 external_mini_cluster.cc:1468] Setting key 5eb01a69225eb3086b5aa850a6cb8a5a
I20260203 21:06:47.935904 2916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:47.925934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:59045 (local address 127.0.102.190:40141)
0203 21:06:47.926077 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:47.926081 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:47.926748 (+ 667us) server_negotiation.cc:408] Connection header received
0203 21:06:47.927577 (+ 829us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:47.927580 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:47.927636 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:47.927713 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:47.929385 (+ 1672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:47.929882 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:47.930523 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:47.930705 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:47.932798 (+ 2093us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:47.932817 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:47.932819 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:47.932844 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:47.933973 (+ 1129us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:47.934451 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:47.934454 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:47.934455 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:47.934498 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:47.934773 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:47.934776 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:47.934778 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:47.934922 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:47.935013 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:47.935619 (+ 606us) server_negotiation.cc:300] Negotiation successful
0203 21:06:47.935739 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260203 21:06:47.936489 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:47.926176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:40141 (local address 127.0.102.129:59045)
0203 21:06:47.926613 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:47.926647 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:47.927374 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:47.927886 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:47.927894 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:47.928272 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:47.929226 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:47.929243 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:47.929999 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:47.930003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:47.930404 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:47.930411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:47.930629 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:47.931215 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:47.931240 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:47.932621 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:47.934082 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:47.934088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:47.934099 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:47.934357 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:47.934578 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:47.934581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:47.934583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:47.934679 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:47.935053 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:47.935059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:47.935342 (+ 283us) client_negotiation.cc:769] Sending connection context
0203 21:06:47.935570 (+ 228us) client_negotiation.cc:240] Negotiation successful
0203 21:06:47.935801 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":118,"threads_started":1}
I20260203 21:06:47.937599 3070 heartbeater.cc:344] Connected to a master server at 127.0.102.190:40141
I20260203 21:06:47.937815 3070 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:47.938278 3070 heartbeater.cc:507] Master 127.0.102.190:40141 requested a full tablet report, sending...
I20260203 21:06:47.939581 2877 ts_manager.cc:194] Registered new tserver with Master: 4737be95d6dd452596f0e6bb7a87eb78 (127.0.102.129:46733)
I20260203 21:06:47.940724 2877 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:59045
WARNING: no policy specified for kudu/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:47.975210 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:40141
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:43053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:48.077560 3077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:48.077795 3077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:48.077883 3077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:48.081236 3077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:48.081315 3077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:48.081430 3077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:06:48.085626 3077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:43053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:40141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:48.086700 3077 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:48.087555 3077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:48.094259 3082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:48.094249 3085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:48.094249 3083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:48.094866 3077 server_base.cc:1060] running on GCE node
I20260203 21:06:48.095206 3077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:48.095717 3077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:48.096854 3077 hybrid_clock.cc:648] HybridClock initialized: now 1770152808096832 us; error 36 us; skew 500 ppm
Feb 03 21:06:48 dist-test-slave-j889 krb5kdc[2829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:48.099205 3077 init.cc:377] Logged in from keytab as kudu/127.0.102.130@KRBTEST.COM (short username kudu)
I20260203 21:06:48.100320 3077 webserver.cc:492] Webserver started at http://127.0.102.130:42935/ using document root <none> and password file <none>
I20260203 21:06:48.100885 3077 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:48.100937 3077 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:48.101107 3077 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:48.102718 3077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "00998485565f43a485a6cc6a5f979118"
format_stamp: "Formatted at 2026-02-03 21:06:48 on dist-test-slave-j889"
server_key: "f1ae6a43e24c03042a8ab43cb42ca431"
server_key_iv: "19a328ed5eae31427a3c6da7b92d74d0"
server_key_version: "encryptionkey@0"
I20260203 21:06:48.103155 3077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "00998485565f43a485a6cc6a5f979118"
format_stamp: "Formatted at 2026-02-03 21:06:48 on dist-test-slave-j889"
server_key: "f1ae6a43e24c03042a8ab43cb42ca431"
server_key_iv: "19a328ed5eae31427a3c6da7b92d74d0"
server_key_version: "encryptionkey@0"
I20260203 21:06:48.106310 3077 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260203 21:06:48.108363 3092 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:48.109355 3077 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260203 21:06:48.109450 3077 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "00998485565f43a485a6cc6a5f979118"
format_stamp: "Formatted at 2026-02-03 21:06:48 on dist-test-slave-j889"
server_key: "f1ae6a43e24c03042a8ab43cb42ca431"
server_key_iv: "19a328ed5eae31427a3c6da7b92d74d0"
server_key_version: "encryptionkey@0"
I20260203 21:06:48.109531 3077 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:48.125169 3077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:48.127840 3077 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:48.128016 3077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:48.128575 3077 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:48.129487 3077 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:48.129536 3077 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:48.129590 3077 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:48.129619 3077 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:48.138357 3077 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:44015
I20260203 21:06:48.138392 3205 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:44015 every 8 connection(s)
I20260203 21:06:48.139264 3077 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:06:48.140388 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 3077
I20260203 21:06:48.140492 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:06:48.140744 410 external_mini_cluster.cc:1468] Setting key db844069c866292e00a09e169e068e1b
Feb 03 21:06:48 dist-test-slave-j889 krb5kdc[2829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.131@KRBTEST.COM" created.
I20260203 21:06:48.150763 2916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:48.140978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:47901 (local address 127.0.102.190:40141)
0203 21:06:48.141125 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:48.141130 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:48.141853 (+ 723us) server_negotiation.cc:408] Connection header received
0203 21:06:48.142687 (+ 834us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:48.142691 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:48.142742 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:48.142827 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:48.144392 (+ 1565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:48.144919 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:48.145571 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:48.145746 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:48.147731 (+ 1985us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:48.147749 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:48.147751 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:48.147775 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:48.149050 (+ 1275us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:48.149499 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:48.149502 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:48.149504 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:48.149546 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:48.149799 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:48.149802 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:48.149804 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:48.149939 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:48.150020 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:48.150517 (+ 497us) server_negotiation.cc:300] Negotiation successful
0203 21:06:48.150639 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260203 21:06:48.151535 3208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:48.141262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:40141 (local address 127.0.102.130:47901)
0203 21:06:48.141719 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:48.141762 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:48.142487 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:48.142958 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:48.142966 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:48.143338 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:48.144232 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:48.144245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:48.145033 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:48.145037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:48.145451 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:48.145458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:48.145676 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:48.146218 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:48.146244 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:48.147586 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:48.149165 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:48.149170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:48.149181 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:48.149399 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:48.149630 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:48.149633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:48.149634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:48.149715 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:48.150035 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:48.150041 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:48.150252 (+ 211us) client_negotiation.cc:769] Sending connection context
0203 21:06:48.150436 (+ 184us) client_negotiation.cc:240] Negotiation successful
0203 21:06:48.150674 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":111,"threads_started":1}
I20260203 21:06:48.152602 3206 heartbeater.cc:344] Connected to a master server at 127.0.102.190:40141
I20260203 21:06:48.152846 3206 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:48.153342 3206 heartbeater.cc:507] Master 127.0.102.190:40141 requested a full tablet report, sending...
I20260203 21:06:48.154333 2877 ts_manager.cc:194] Registered new tserver with Master: 00998485565f43a485a6cc6a5f979118 (127.0.102.130:44015)
I20260203 21:06:48.154878 2877 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.130@KRBTEST.COM'} at 127.0.102.130:47901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:48.183831 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:40141
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:43053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:48.287554 3213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:48.287781 3213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:48.287860 3213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:48.291144 3213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:48.291224 3213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:48.291332 3213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:06:48.295625 3213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:43053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:40141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:48.296828 3213 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:48.297677 3213 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:48.303889 3218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:48.303890 3221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:48.303993 3219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:48.304045 3213 server_base.cc:1060] running on GCE node
I20260203 21:06:48.304445 3213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:48.304992 3213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:48.306134 3213 hybrid_clock.cc:648] HybridClock initialized: now 1770152808306118 us; error 29 us; skew 500 ppm
Feb 03 21:06:48 dist-test-slave-j889 krb5kdc[2829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:48.308503 3213 init.cc:377] Logged in from keytab as kudu/127.0.102.131@KRBTEST.COM (short username kudu)
I20260203 21:06:48.309459 3213 webserver.cc:492] Webserver started at http://127.0.102.131:38401/ using document root <none> and password file <none>
I20260203 21:06:48.309955 3213 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:48.310003 3213 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:48.310164 3213 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:48.311764 3213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "8ece9f89580047ad93172147bb1a18d7"
format_stamp: "Formatted at 2026-02-03 21:06:48 on dist-test-slave-j889"
server_key: "bc019c161e8d7e919914b291da481042"
server_key_iv: "c711be4c5c0b0fdb47e3eccf757aa583"
server_key_version: "encryptionkey@0"
I20260203 21:06:48.312182 3213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "8ece9f89580047ad93172147bb1a18d7"
format_stamp: "Formatted at 2026-02-03 21:06:48 on dist-test-slave-j889"
server_key: "bc019c161e8d7e919914b291da481042"
server_key_iv: "c711be4c5c0b0fdb47e3eccf757aa583"
server_key_version: "encryptionkey@0"
I20260203 21:06:48.315266 3213 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260203 21:06:48.317246 3228 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:48.318161 3213 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:06:48.318253 3213 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "8ece9f89580047ad93172147bb1a18d7"
format_stamp: "Formatted at 2026-02-03 21:06:48 on dist-test-slave-j889"
server_key: "bc019c161e8d7e919914b291da481042"
server_key_iv: "c711be4c5c0b0fdb47e3eccf757aa583"
server_key_version: "encryptionkey@0"
I20260203 21:06:48.318331 3213 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:48.332414 3213 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:48.335328 3213 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:48.335510 3213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:48.336041 3213 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:48.336910 3213 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:48.336957 3213 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:48.337008 3213 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:48.337036 3213 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:48.345312 3213 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:33329
I20260203 21:06:48.345367 3341 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:33329 every 8 connection(s)
I20260203 21:06:48.346197 3213 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
I20260203 21:06:48.349080 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 3213
I20260203 21:06:48.349196 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:06:48.349447 410 external_mini_cluster.cc:1468] Setting key 962bb63c34a754bbb33e98bbf0623a68
Feb 03 21:06:48 dist-test-slave-j889 krb5kdc[2829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:48.357676 2916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:48.347801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:60281 (local address 127.0.102.190:40141)
0203 21:06:48.347919 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:48.347923 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:48.348602 (+ 679us) server_negotiation.cc:408] Connection header received
0203 21:06:48.349609 (+ 1007us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:48.349613 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:48.349663 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:48.349744 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:48.351604 (+ 1860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:48.352088 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:48.352787 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:48.352938 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:48.354831 (+ 1893us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:48.354845 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:48.354847 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:48.354872 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:48.355989 (+ 1117us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:48.356440 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:48.356442 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:48.356444 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:48.356479 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:48.356802 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:48.356804 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:48.356805 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:48.356944 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:48.357017 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:48.357467 (+ 450us) server_negotiation.cc:300] Negotiation successful
0203 21:06:48.357557 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260203 21:06:48.358275 3344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:48.348018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:40141 (local address 127.0.102.131:60281)
0203 21:06:48.348455 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:48.348489 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:48.349388 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:48.349897 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:48.349908 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:48.350404 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:48.351480 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:48.351494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:48.352196 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:48.352199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:48.352663 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:48.352673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:48.352841 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:48.353363 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:48.353387 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:48.354701 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:48.356102 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:48.356106 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:48.356115 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:48.356354 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:48.356576 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:48.356579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:48.356580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:48.356723 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:48.357052 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:48.357057 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:48.357268 (+ 211us) client_negotiation.cc:769] Sending connection context
0203 21:06:48.357432 (+ 164us) client_negotiation.cc:240] Negotiation successful
0203 21:06:48.357649 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":96,"threads_started":1}
I20260203 21:06:48.359230 3342 heartbeater.cc:344] Connected to a master server at 127.0.102.190:40141
I20260203 21:06:48.359437 3342 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:48.359889 3342 heartbeater.cc:507] Master 127.0.102.190:40141 requested a full tablet report, sending...
I20260203 21:06:48.360778 2877 ts_manager.cc:194] Registered new tserver with Master: 8ece9f89580047ad93172147bb1a18d7 (127.0.102.131:33329)
I20260203 21:06:48.361271 2877 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.131@KRBTEST.COM'} at 127.0.102.131:60281
I20260203 21:06:48.362974 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260203 21:06:48.373484 2916 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:06:48.372315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49404 (local address 127.0.102.190:40141)
0203 21:06:48.372450 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:48.372454 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:48.372526 (+ 72us) server_negotiation.cc:408] Connection header received
0203 21:06:48.372657 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:48.372661 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:48.372713 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:48.372787 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:48.373379 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49404: BlockingRecv error: recv got EOF from 127.0.0.1:49404 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260203 21:06:48.374769 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 2941
I20260203 21:06:48.379679 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 3077
I20260203 21:06:48.384002 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 3213
I20260203 21:06:48.388440 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 2845
2026-02-03T21:06:48Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2886 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:06:48 dist-test-slave-j889 krb5kdc[3354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:06:48 dist-test-slave-j889 krb5kdc[3354](info): set up 2 sockets
Feb 03 21:06:48 dist-test-slave-j889 krb5kdc[3354](info): commencing operation
krb5kdc: starting...
W20260203 21:06:50.212695 410 mini_kdc.cc:121] Time spent starting KDC: real 1.804s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:06:50 dist-test-slave-j889 krb5kdc[3354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:06:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:06:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:50.337988 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:37149
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:37149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:50.441098 3370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:50.441380 3370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:50.441479 3370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:50.445118 3370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:06:50.445171 3370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:50.445194 3370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:50.445210 3370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:06:50.445226 3370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:06:50.449280 3370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:37149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:37149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:50.450214 3370 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:50.450996 3370 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:50.456248 3378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:50.456274 3376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:50.456305 3375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:50.456650 3370 server_base.cc:1060] running on GCE node
I20260203 21:06:50.457218 3370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:50.458125 3370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:50.459299 3370 hybrid_clock.cc:648] HybridClock initialized: now 1770152810459272 us; error 40 us; skew 500 ppm
Feb 03 21:06:50 dist-test-slave-j889 krb5kdc[3354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:50.461804 3370 init.cc:377] Logged in from keytab as kudu/127.0.102.190@KRBTEST.COM (short username kudu)
I20260203 21:06:50.462801 3370 webserver.cc:492] Webserver started at http://127.0.102.190:39075/ using document root <none> and password file <none>
I20260203 21:06:50.463428 3370 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:50.463485 3370 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:50.463697 3370 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:50.465346 3370 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "1c4625555acd4d208607a62d117eec33"
format_stamp: "Formatted at 2026-02-03 21:06:50 on dist-test-slave-j889"
server_key: "dcd0c8e377bfa2119a73ab24969d908a"
server_key_iv: "b4451996fa011af377dd127e6426eaac"
server_key_version: "encryptionkey@0"
I20260203 21:06:50.465767 3370 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "1c4625555acd4d208607a62d117eec33"
format_stamp: "Formatted at 2026-02-03 21:06:50 on dist-test-slave-j889"
server_key: "dcd0c8e377bfa2119a73ab24969d908a"
server_key_iv: "b4451996fa011af377dd127e6426eaac"
server_key_version: "encryptionkey@0"
I20260203 21:06:50.468916 3370 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260203 21:06:50.470948 3385 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:50.472025 3370 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:06:50.472251 3370 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "1c4625555acd4d208607a62d117eec33"
format_stamp: "Formatted at 2026-02-03 21:06:50 on dist-test-slave-j889"
server_key: "dcd0c8e377bfa2119a73ab24969d908a"
server_key_iv: "b4451996fa011af377dd127e6426eaac"
server_key_version: "encryptionkey@0"
I20260203 21:06:50.472360 3370 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:50.498646 3370 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:50.501598 3370 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:50.501783 3370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:50.508842 3437 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:37149 every 8 connection(s)
I20260203 21:06:50.508852 3370 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:37149
I20260203 21:06:50.509876 3370 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:06:50.512486 3438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:50.513140 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 3370
I20260203 21:06:50.513250 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:06:50.513504 410 external_mini_cluster.cc:1468] Setting key f6fae2c95d95883bb059810ebcb7baa0
I20260203 21:06:50.518357 3438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33: Bootstrap starting.
Feb 03 21:06:50 dist-test-slave-j889 krb5kdc[3354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:50.520462 3438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:50.521173 3438 log.cc:826] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:50.522828 3438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33: No bootstrap required, opened a new log
I20260203 21:06:50.524973 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:50.514680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57780 (local address 127.0.102.190:37149)
0203 21:06:50.515092 (+ 412us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:50.515101 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:50.515133 (+ 32us) server_negotiation.cc:408] Connection header received
0203 21:06:50.515720 (+ 587us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:50.515738 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:50.516009 (+ 271us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:50.516298 (+ 289us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:50.517209 (+ 911us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:50.517920 (+ 711us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:50.518574 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:50.518832 (+ 258us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:50.520765 (+ 1933us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:50.520782 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:50.520792 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:50.520823 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:50.522704 (+ 1881us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:50.523107 (+ 403us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:50.523111 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:50.523116 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:50.523187 (+ 71us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:50.523400 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:50.523404 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:50.523406 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:50.523720 (+ 314us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:50.523900 (+ 180us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:50.524165 (+ 265us) server_negotiation.cc:300] Negotiation successful
0203 21:06:50.524385 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":136,"threads_started":1}
I20260203 21:06:50.525266 3438 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4625555acd4d208607a62d117eec33" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 37149 } }
I20260203 21:06:50.525434 3438 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:50.525496 3438 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c4625555acd4d208607a62d117eec33, State: Initialized, Role: FOLLOWER
I20260203 21:06:50.525902 3438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4625555acd4d208607a62d117eec33" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 37149 } }
I20260203 21:06:50.526041 3438 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:06:50.526098 3438 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:06:50.526176 3438 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:50.527112 3438 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4625555acd4d208607a62d117eec33" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 37149 } }
I20260203 21:06:50.527390 3438 leader_election.cc:304] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c4625555acd4d208607a62d117eec33; no voters:
I20260203 21:06:50.527601 3438 leader_election.cc:290] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:06:50.527720 3443 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:50.527906 3443 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [term 1 LEADER]: Becoming Leader. State: Replica: 1c4625555acd4d208607a62d117eec33, State: Running, Role: LEADER
I20260203 21:06:50.528184 3443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4625555acd4d208607a62d117eec33" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 37149 } }
I20260203 21:06:50.528566 3438 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:06:50.529678 3445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c4625555acd4d208607a62d117eec33. Latest consensus state: current_term: 1 leader_uuid: "1c4625555acd4d208607a62d117eec33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4625555acd4d208607a62d117eec33" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 37149 } } }
I20260203 21:06:50.529786 3445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [sys.catalog]: This master's current role is: LEADER
I20260203 21:06:50.529711 3444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c4625555acd4d208607a62d117eec33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4625555acd4d208607a62d117eec33" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 37149 } } }
I20260203 21:06:50.529960 3444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33 [sys.catalog]: This master's current role is: LEADER
W20260203 21:06:50.533030 3458 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260203 21:06:50.533093 3458 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260203 21:06:50.533144 3452 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:06:50.533842 3452 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:06:50.538436 3452 catalog_manager.cc:1357] Generated new cluster ID: 73d1f2dbac954cbaa9bd61c261f48f2e
I20260203 21:06:50.538494 3452 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:06:50.558390 3452 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:06:50.559315 3452 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:06:50.565143 3452 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 1c4625555acd4d208607a62d117eec33: Generated new TSK 0
I20260203 21:06:50.565905 3452 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:50.611162 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:37149
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:50.712872 3466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:50.713076 3466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:50.713140 3466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:50.716322 3466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:50.716375 3466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:50.716476 3466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:06:50.721197 3466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:37149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:50.722159 3466 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:50.722944 3466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:50.728955 3474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:50.728956 3471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:50.729024 3472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:50.729187 3466 server_base.cc:1060] running on GCE node
I20260203 21:06:50.729509 3466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:50.730053 3466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:50.731195 3466 hybrid_clock.cc:648] HybridClock initialized: now 1770152810731185 us; error 28 us; skew 500 ppm
Feb 03 21:06:50 dist-test-slave-j889 krb5kdc[3354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:50.733598 3466 init.cc:377] Logged in from keytab as kudu/127.0.102.129@KRBTEST.COM (short username kudu)
I20260203 21:06:50.734578 3466 webserver.cc:492] Webserver started at http://127.0.102.129:33735/ using document root <none> and password file <none>
I20260203 21:06:50.735097 3466 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:50.735146 3466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:50.735333 3466 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:50.737035 3466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "faa0fb37670d4efd9270e4d4e5573deb"
format_stamp: "Formatted at 2026-02-03 21:06:50 on dist-test-slave-j889"
server_key: "55c1e741b5c57e481f253aeb8de3acfd"
server_key_iv: "36bc40e6d95fd87166e95c96bd959d84"
server_key_version: "encryptionkey@0"
I20260203 21:06:50.737483 3466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "faa0fb37670d4efd9270e4d4e5573deb"
format_stamp: "Formatted at 2026-02-03 21:06:50 on dist-test-slave-j889"
server_key: "55c1e741b5c57e481f253aeb8de3acfd"
server_key_iv: "36bc40e6d95fd87166e95c96bd959d84"
server_key_version: "encryptionkey@0"
I20260203 21:06:50.740718 3466 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260203 21:06:50.742834 3481 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:50.743783 3466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:06:50.743906 3466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "faa0fb37670d4efd9270e4d4e5573deb"
format_stamp: "Formatted at 2026-02-03 21:06:50 on dist-test-slave-j889"
server_key: "55c1e741b5c57e481f253aeb8de3acfd"
server_key_iv: "36bc40e6d95fd87166e95c96bd959d84"
server_key_version: "encryptionkey@0"
I20260203 21:06:50.743991 3466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:50.758368 3466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:50.764415 3466 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:50.764604 3466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:50.765178 3466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:50.766047 3466 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:50.766099 3466 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:50.766167 3466 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:50.766198 3466 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:50.775569 3466 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:38233
I20260203 21:06:50.775590 3594 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:38233 every 8 connection(s)
I20260203 21:06:50.776612 3466 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
Feb 03 21:06:50 dist-test-slave-j889 krb5kdc[3354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:50.786053 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 3466
I20260203 21:06:50.786146 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:06:50.786371 410 external_mini_cluster.cc:1468] Setting key 7febcd6b9fef5462350f10c1a7c986d7
I20260203 21:06:50.788535 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:50.778317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:35735 (local address 127.0.102.190:37149)
0203 21:06:50.778464 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:50.778470 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:50.779154 (+ 684us) server_negotiation.cc:408] Connection header received
0203 21:06:50.779945 (+ 791us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:50.779948 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:50.779999 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:50.780080 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:50.781664 (+ 1584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:50.782204 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:50.782879 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:50.783024 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:50.784942 (+ 1918us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:50.784962 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:50.784964 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:50.784988 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:50.786242 (+ 1254us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:50.786769 (+ 527us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:50.786773 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:50.786775 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:50.786829 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:50.787505 (+ 676us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:50.787507 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:50.787508 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:50.787710 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:50.787804 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:50.788323 (+ 519us) server_negotiation.cc:300] Negotiation successful
0203 21:06:50.788424 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260203 21:06:50.789309 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:50.778574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:37149 (local address 127.0.102.129:35735)
0203 21:06:50.779017 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:50.779051 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:50.779752 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:50.780216 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:50.780224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:50.780599 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:50.781532 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:50.781546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:50.782309 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:50.782312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:50.782769 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:50.782777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:50.782934 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:50.783431 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:50.783454 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:50.784815 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:50.786345 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:50.786350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:50.786360 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:50.786636 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:50.786919 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:50.786922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:50.786924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:50.787028 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:50.787824 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:50.787830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:50.788086 (+ 256us) client_negotiation.cc:769] Sending connection context
0203 21:06:50.788298 (+ 212us) client_negotiation.cc:240] Negotiation successful
0203 21:06:50.788546 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":107,"threads_started":1}
I20260203 21:06:50.790349 3595 heartbeater.cc:344] Connected to a master server at 127.0.102.190:37149
I20260203 21:06:50.790558 3595 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:50.791039 3595 heartbeater.cc:507] Master 127.0.102.190:37149 requested a full tablet report, sending...
I20260203 21:06:50.792287 3402 ts_manager.cc:194] Registered new tserver with Master: faa0fb37670d4efd9270e4d4e5573deb (127.0.102.129:38233)
I20260203 21:06:50.793507 3402 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:35735
WARNING: no policy specified for kudu/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:50.827384 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:37149
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:50.928007 3602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:50.928198 3602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:50.928273 3602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:50.931535 3602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:50.931602 3602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:50.931696 3602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:06:50.935989 3602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:37149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:50.937189 3602 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:50.938035 3602 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:50.944175 3610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:50.944178 3608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:50.944165 3607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:50.944545 3602 server_base.cc:1060] running on GCE node
I20260203 21:06:50.944911 3602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:50.945415 3602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:50.946563 3602 hybrid_clock.cc:648] HybridClock initialized: now 1770152810946538 us; error 37 us; skew 500 ppm
Feb 03 21:06:50 dist-test-slave-j889 krb5kdc[3354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:50.948968 3602 init.cc:377] Logged in from keytab as kudu/127.0.102.130@KRBTEST.COM (short username kudu)
I20260203 21:06:50.949896 3602 webserver.cc:492] Webserver started at http://127.0.102.130:45539/ using document root <none> and password file <none>
I20260203 21:06:50.950377 3602 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:50.950421 3602 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:50.950562 3602 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:50.952123 3602 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "bf4eef79f1eb4202a323e6058e500acc"
format_stamp: "Formatted at 2026-02-03 21:06:50 on dist-test-slave-j889"
server_key: "afcf9be31a55b0e1c876863120cab6d5"
server_key_iv: "e7fcf69563c30d3097feec7a60f9ddec"
server_key_version: "encryptionkey@0"
I20260203 21:06:50.952539 3602 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "bf4eef79f1eb4202a323e6058e500acc"
format_stamp: "Formatted at 2026-02-03 21:06:50 on dist-test-slave-j889"
server_key: "afcf9be31a55b0e1c876863120cab6d5"
server_key_iv: "e7fcf69563c30d3097feec7a60f9ddec"
server_key_version: "encryptionkey@0"
I20260203 21:06:50.955693 3602 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260203 21:06:50.957736 3617 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:50.958670 3602 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260203 21:06:50.958767 3602 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "bf4eef79f1eb4202a323e6058e500acc"
format_stamp: "Formatted at 2026-02-03 21:06:50 on dist-test-slave-j889"
server_key: "afcf9be31a55b0e1c876863120cab6d5"
server_key_iv: "e7fcf69563c30d3097feec7a60f9ddec"
server_key_version: "encryptionkey@0"
I20260203 21:06:50.958851 3602 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:50.982846 3602 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:50.985364 3602 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:50.985530 3602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:50.986063 3602 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:50.986912 3602 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:50.986958 3602 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:50.987018 3602 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:50.987047 3602 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:50.995322 3602 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:44217
I20260203 21:06:50.995354 3730 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:44217 every 8 connection(s)
I20260203 21:06:50.996217 3602 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:06:51.001943 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 3602
I20260203 21:06:51.002063 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:06:51.002317 410 external_mini_cluster.cc:1468] Setting key 85e5b1c9307f9acbe25cac1b0ae09cff
Feb 03 21:06:50 dist-test-slave-j889 krb5kdc[3354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:51.008157 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:50.997805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:45261 (local address 127.0.102.190:37149)
0203 21:06:50.997943 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:50.997947 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:50.998564 (+ 617us) server_negotiation.cc:408] Connection header received
0203 21:06:50.999321 (+ 757us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:50.999324 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:50.999371 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:50.999440 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:51.001048 (+ 1608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:51.001536 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:51.002297 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:51.002466 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:51.004871 (+ 2405us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:51.004888 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:51.004890 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:51.004916 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:51.006337 (+ 1421us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:51.006773 (+ 436us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:51.006777 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:51.006779 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:51.006829 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:51.007119 (+ 290us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:51.007123 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:51.007125 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:51.007312 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:51.007408 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:51.007867 (+ 459us) server_negotiation.cc:300] Negotiation successful
0203 21:06:51.007987 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260203 21:06:51.008749 3733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:50.998046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:37149 (local address 127.0.102.130:45261)
0203 21:06:50.998428 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:50.998462 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:50.999152 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:50.999572 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:50.999579 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:50.999961 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:51.000903 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:51.000917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:51.001629 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:51.001631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:51.002194 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:51.002204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:51.002415 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:51.003039 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:51.003075 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:51.004729 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:51.006446 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:51.006451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:51.006460 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:51.006682 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:51.006923 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:51.006926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:51.006928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:51.007034 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:51.007404 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:51.007408 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:51.007629 (+ 221us) client_negotiation.cc:769] Sending connection context
0203 21:06:51.007796 (+ 167us) client_negotiation.cc:240] Negotiation successful
0203 21:06:51.008003 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":106,"threads_started":1}
I20260203 21:06:51.009773 3731 heartbeater.cc:344] Connected to a master server at 127.0.102.190:37149
I20260203 21:06:51.010000 3731 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:51.010489 3731 heartbeater.cc:507] Master 127.0.102.190:37149 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.102.131@KRBTEST.COM; defaulting to no policy
I20260203 21:06:51.011394 3402 ts_manager.cc:194] Registered new tserver with Master: bf4eef79f1eb4202a323e6058e500acc (127.0.102.130:44217)
I20260203 21:06:51.011934 3402 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.130@KRBTEST.COM'} at 127.0.102.130:45261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:51.044824 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:37149
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:51.148511 3738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:51.148770 3738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:51.148850 3738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:51.152103 3738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:51.152169 3738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:51.152268 3738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:06:51.156455 3738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:37149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:51.157509 3738 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:51.158324 3738 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:51.164508 3744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:51.164572 3743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:51.164582 3746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:51.164821 3738 server_base.cc:1060] running on GCE node
I20260203 21:06:51.165225 3738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:51.165731 3738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:51.166877 3738 hybrid_clock.cc:648] HybridClock initialized: now 1770152811166844 us; error 54 us; skew 500 ppm
Feb 03 21:06:51 dist-test-slave-j889 krb5kdc[3354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:51.169294 3738 init.cc:377] Logged in from keytab as kudu/127.0.102.131@KRBTEST.COM (short username kudu)
I20260203 21:06:51.170261 3738 webserver.cc:492] Webserver started at http://127.0.102.131:36661/ using document root <none> and password file <none>
I20260203 21:06:51.170771 3738 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:51.170821 3738 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:51.171000 3738 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:51.172621 3738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "b729c9a2579641a288f010121f9e89ba"
format_stamp: "Formatted at 2026-02-03 21:06:51 on dist-test-slave-j889"
server_key: "01a325363d73eaae8544139aa23bd320"
server_key_iv: "3095ddb4046fd4d8c3ff4fbb319b65f2"
server_key_version: "encryptionkey@0"
I20260203 21:06:51.173141 3738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "b729c9a2579641a288f010121f9e89ba"
format_stamp: "Formatted at 2026-02-03 21:06:51 on dist-test-slave-j889"
server_key: "01a325363d73eaae8544139aa23bd320"
server_key_iv: "3095ddb4046fd4d8c3ff4fbb319b65f2"
server_key_version: "encryptionkey@0"
I20260203 21:06:51.176395 3738 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260203 21:06:51.178601 3753 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:51.179603 3738 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:06:51.179700 3738 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "b729c9a2579641a288f010121f9e89ba"
format_stamp: "Formatted at 2026-02-03 21:06:51 on dist-test-slave-j889"
server_key: "01a325363d73eaae8544139aa23bd320"
server_key_iv: "3095ddb4046fd4d8c3ff4fbb319b65f2"
server_key_version: "encryptionkey@0"
I20260203 21:06:51.179785 3738 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:51.191530 3738 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:51.193933 3738 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:51.194094 3738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:51.194577 3738 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:51.195356 3738 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:51.195401 3738 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:51.195448 3738 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:51.195477 3738 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:51.203719 3738 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:39069
I20260203 21:06:51.203765 3866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:39069 every 8 connection(s)
I20260203 21:06:51.204555 3738 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
I20260203 21:06:51.209723 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 3738
I20260203 21:06:51.209843 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:06:51.210067 410 external_mini_cluster.cc:1468] Setting key 2b890f1c1759c084af6e39b08811f90a
Feb 03 21:06:51 dist-test-slave-j889 krb5kdc[3354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:51.215704 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:51.206193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:42555 (local address 127.0.102.190:37149)
0203 21:06:51.206308 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:51.206312 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:51.206939 (+ 627us) server_negotiation.cc:408] Connection header received
0203 21:06:51.207714 (+ 775us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:51.207718 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:51.207784 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:51.207854 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:51.209499 (+ 1645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:51.210067 (+ 568us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:51.210785 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:51.210963 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:51.213040 (+ 2077us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:51.213057 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:51.213059 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:51.213086 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:51.214186 (+ 1100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:51.214609 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:51.214612 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:51.214613 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:51.214651 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:51.214858 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:51.214860 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:51.214861 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:51.214999 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:51.215068 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:51.215491 (+ 423us) server_negotiation.cc:300] Negotiation successful
0203 21:06:51.215585 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260203 21:06:51.216332 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:51.206417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:37149 (local address 127.0.102.131:42555)
0203 21:06:51.206812 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:51.206846 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:51.207533 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:51.208032 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:51.208039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:51.208410 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:51.209352 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:51.209367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:51.210181 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:51.210185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:51.210659 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:51.210667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:51.210863 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:51.211416 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:51.211440 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:51.212903 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:51.214296 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:51.214301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:51.214310 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:51.214534 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:51.214712 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:51.214714 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:51.214715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:51.214794 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:51.215069 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:51.215073 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:51.215303 (+ 230us) client_negotiation.cc:769] Sending connection context
0203 21:06:51.215465 (+ 162us) client_negotiation.cc:240] Negotiation successful
0203 21:06:51.215669 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":108,"threads_started":1}
I20260203 21:06:51.217298 3867 heartbeater.cc:344] Connected to a master server at 127.0.102.190:37149
I20260203 21:06:51.217526 3867 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:51.218017 3867 heartbeater.cc:507] Master 127.0.102.190:37149 requested a full tablet report, sending...
I20260203 21:06:51.218921 3402 ts_manager.cc:194] Registered new tserver with Master: b729c9a2579641a288f010121f9e89ba (127.0.102.131:39069)
I20260203 21:06:51.219442 3402 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.131@KRBTEST.COM'} at 127.0.102.131:42555
I20260203 21:06:51.223474 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260203 21:06:51.347080 3441 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:06:51.343668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.0.102.190:37149)
0203 21:06:51.343800 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:51.343804 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:51.344727 (+ 923us) server_negotiation.cc:408] Connection header received
0203 21:06:51.345588 (+ 861us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:51.345590 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:51.345639 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:51.345711 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:51.346938 (+ 1227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57796: BlockingRecv error: recv got EOF from 127.0.0.1:57796 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260203 21:06:51.348532 3441 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:06:51.347704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.0.102.190:37149)
0203 21:06:51.347831 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:51.347835 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:51.347849 (+ 14us) server_negotiation.cc:408] Connection header received
0203 21:06:51.347937 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:51.347940 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:51.347991 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:51.348054 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:51.348461 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57804: BlockingRecv error: recv got EOF from 127.0.0.1:57804 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260203 21:06:51.349588 3441 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:06:51.348780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.0.102.190:37149)
0203 21:06:51.348871 (+ 91us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:51.348875 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:51.349013 (+ 138us) server_negotiation.cc:408] Connection header received
0203 21:06:51.349102 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:51.349105 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:51.349143 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:51.349205 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:51.349511 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57812: BlockingRecv error: recv got EOF from 127.0.0.1:57812 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260203 21:06:51.353082 3441 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:06:51.352337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57824 (local address 127.0.102.190:37149)
0203 21:06:51.352448 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:51.352452 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:51.352465 (+ 13us) server_negotiation.cc:408] Connection header received
0203 21:06:51.352550 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:51.352553 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:51.352586 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:51.352669 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:51.353014 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57824: BlockingRecv error: recv got EOF from 127.0.0.1:57824 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260203 21:06:51.360152 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 3466
I20260203 21:06:51.365306 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 3602
I20260203 21:06:51.369690 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 3738
I20260203 21:06:51.374090 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 3370
2026-02-03T21:06:51Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (2985 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:06:51 dist-test-slave-j889 krb5kdc[3890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:06:51 dist-test-slave-j889 krb5kdc[3890](info): set up 2 sockets
Feb 03 21:06:51 dist-test-slave-j889 krb5kdc[3890](info): commencing operation
krb5kdc: starting...
W20260203 21:06:53.228902 410 mini_kdc.cc:121] Time spent starting KDC: real 1.835s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:06:53 dist-test-slave-j889 krb5kdc[3890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:06:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:06:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:53.351344 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:43591
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:46239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:43591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:53.456123 3906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:53.456357 3906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:53.456430 3906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:53.459659 3906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:06:53.459728 3906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:53.459753 3906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:53.459776 3906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:06:53.459815 3906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:06:53.463955 3906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:43591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:43591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:53.465032 3906 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:53.465847 3906 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:53.471529 3911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:53.471536 3912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:53.471537 3914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:53.471735 3906 server_base.cc:1060] running on GCE node
I20260203 21:06:53.472225 3906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:53.473132 3906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:53.474308 3906 hybrid_clock.cc:648] HybridClock initialized: now 1770152813474289 us; error 29 us; skew 500 ppm
Feb 03 21:06:53 dist-test-slave-j889 krb5kdc[3890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:53.476789 3906 init.cc:377] Logged in from keytab as kudu/127.0.102.190@KRBTEST.COM (short username kudu)
I20260203 21:06:53.477792 3906 webserver.cc:492] Webserver started at http://127.0.102.190:45343/ using document root <none> and password file <none>
I20260203 21:06:53.478300 3906 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:53.478348 3906 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:53.478514 3906 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:53.480104 3906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "2d10473f06944f80a12b8b6a45274cc9"
format_stamp: "Formatted at 2026-02-03 21:06:53 on dist-test-slave-j889"
server_key: "13ebbc5256c62a4d1c6e9ede1ca93a12"
server_key_iv: "dbe48c625f8dc510a37dd665d72860d0"
server_key_version: "encryptionkey@0"
I20260203 21:06:53.480623 3906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "2d10473f06944f80a12b8b6a45274cc9"
format_stamp: "Formatted at 2026-02-03 21:06:53 on dist-test-slave-j889"
server_key: "13ebbc5256c62a4d1c6e9ede1ca93a12"
server_key_iv: "dbe48c625f8dc510a37dd665d72860d0"
server_key_version: "encryptionkey@0"
I20260203 21:06:53.483811 3906 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260203 21:06:53.485867 3921 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:53.486783 3906 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260203 21:06:53.486884 3906 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "2d10473f06944f80a12b8b6a45274cc9"
format_stamp: "Formatted at 2026-02-03 21:06:53 on dist-test-slave-j889"
server_key: "13ebbc5256c62a4d1c6e9ede1ca93a12"
server_key_iv: "dbe48c625f8dc510a37dd665d72860d0"
server_key_version: "encryptionkey@0"
I20260203 21:06:53.486966 3906 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:53.504118 3906 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:53.506978 3906 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:53.507164 3906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:53.514195 3906 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:43591
I20260203 21:06:53.514196 3973 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:43591 every 8 connection(s)
I20260203 21:06:53.515172 3906 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:06:53.516151 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 3906
I20260203 21:06:53.516261 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:06:53.516508 410 external_mini_cluster.cc:1468] Setting key 39c196787cec00673644b4f436831038
I20260203 21:06:53.518239 3974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:53.523079 3974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9: Bootstrap starting.
Feb 03 21:06:53 dist-test-slave-j889 krb5kdc[3890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:53.525295 3974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:53.525942 3974 log.cc:826] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:53.527885 3974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9: No bootstrap required, opened a new log
I20260203 21:06:53.528277 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:53.517637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.0.102.190:43591)
0203 21:06:53.518015 (+ 378us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:53.518024 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:53.518063 (+ 39us) server_negotiation.cc:408] Connection header received
0203 21:06:53.519080 (+ 1017us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:53.519100 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:53.519382 (+ 282us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:53.519679 (+ 297us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:53.520537 (+ 858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:53.521330 (+ 793us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:53.521955 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:53.522213 (+ 258us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:53.524099 (+ 1886us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:53.524139 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:53.524149 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:53.524180 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:53.525779 (+ 1599us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:53.526151 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:53.526157 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:53.526162 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:53.526266 (+ 104us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:53.526520 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:53.526522 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:53.526523 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:53.526894 (+ 371us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:53.527101 (+ 207us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:53.527310 (+ 209us) server_negotiation.cc:300] Negotiation successful
0203 21:06:53.527552 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
I20260203 21:06:53.530777 410 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260203 21:06:53.530745 3974 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d10473f06944f80a12b8b6a45274cc9" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43591 } }
I20260203 21:06:53.530925 3974 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:53.530982 3974 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d10473f06944f80a12b8b6a45274cc9, State: Initialized, Role: FOLLOWER
I20260203 21:06:53.531385 3974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d10473f06944f80a12b8b6a45274cc9" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43591 } }
I20260203 21:06:53.531502 3974 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:06:53.531558 3974 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:06:53.531641 3974 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:53.532403 3974 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d10473f06944f80a12b8b6a45274cc9" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43591 } }
I20260203 21:06:53.532747 3974 leader_election.cc:304] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d10473f06944f80a12b8b6a45274cc9; no voters:
I20260203 21:06:53.532992 3974 leader_election.cc:290] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:06:53.533317 3981 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:53.533541 3981 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [term 1 LEADER]: Becoming Leader. State: Replica: 2d10473f06944f80a12b8b6a45274cc9, State: Running, Role: LEADER
I20260203 21:06:53.533850 3981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d10473f06944f80a12b8b6a45274cc9" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43591 } }
I20260203 21:06:53.534121 3974 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:06:53.535364 3980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d10473f06944f80a12b8b6a45274cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d10473f06944f80a12b8b6a45274cc9" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43591 } } }
I20260203 21:06:53.535477 3980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [sys.catalog]: This master's current role is: LEADER
I20260203 21:06:53.535838 3989 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:06:53.535812 3982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d10473f06944f80a12b8b6a45274cc9. Latest consensus state: current_term: 1 leader_uuid: "2d10473f06944f80a12b8b6a45274cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d10473f06944f80a12b8b6a45274cc9" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43591 } } }
I20260203 21:06:53.536083 3982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9 [sys.catalog]: This master's current role is: LEADER
I20260203 21:06:53.539043 3989 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:06:53.544147 3989 catalog_manager.cc:1357] Generated new cluster ID: de64345d119f4099b635f05e93e3afc0
I20260203 21:06:53.544207 3989 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:06:53.547250 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:53.542371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.0.102.190:43591)
0203 21:06:53.542498 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:53.542503 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:53.542734 (+ 231us) server_negotiation.cc:408] Connection header received
0203 21:06:53.542884 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:53.542888 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:53.542940 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:53.542990 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:53.543905 (+ 915us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:53.544699 (+ 794us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:53.545621 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:53.545824 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:53.546297 (+ 473us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:53.546312 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:53.546315 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:06:53.546784 (+ 469us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:53.546824 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:06:53.546841 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:53.546928 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:53.547051 (+ 123us) server_negotiation.cc:300] Negotiation successful
0203 21:06:53.547111 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260203 21:06:53.561333 3989 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:06:53.562517 3989 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:06:53.574901 3989 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 2d10473f06944f80a12b8b6a45274cc9: Generated new TSK 0
I20260203 21:06:53.575780 3989 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260203 21:06:53.580067 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 3906
2026-02-03T21:06:53Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2204 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:06:53 dist-test-slave-j889 krb5kdc[4007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:06:53 dist-test-slave-j889 krb5kdc[4007](info): set up 2 sockets
Feb 03 21:06:53 dist-test-slave-j889 krb5kdc[4007](info): commencing operation
krb5kdc: starting...
W20260203 21:06:55.408593 410 mini_kdc.cc:121] Time spent starting KDC: real 1.810s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:06:55 dist-test-slave-j889 krb5kdc[4007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:06:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:06:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:55.534941 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:44955
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:44955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:55.636461 4023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:55.636723 4023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:55.636787 4023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:55.640012 4023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:06:55.640071 4023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:55.640093 4023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:55.640126 4023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:06:55.640156 4023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:06:55.644381 4023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:44955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:44955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:55.645354 4023 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:55.646152 4023 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:55.651355 4028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:55.651377 4031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:55.651377 4029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:55.651618 4023 server_base.cc:1060] running on GCE node
I20260203 21:06:55.652036 4023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:55.652912 4023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:55.654049 4023 hybrid_clock.cc:648] HybridClock initialized: now 1770152815654032 us; error 33 us; skew 500 ppm
Feb 03 21:06:55 dist-test-slave-j889 krb5kdc[4007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:55.656535 4023 init.cc:377] Logged in from keytab as kudu/127.0.102.190@KRBTEST.COM (short username kudu)
I20260203 21:06:55.657544 4023 webserver.cc:492] Webserver started at http://127.0.102.190:33611/ using document root <none> and password file <none>
I20260203 21:06:55.658063 4023 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:55.658109 4023 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:55.658286 4023 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:55.659993 4023 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "4c580447d1e44ae1b5aba3fbd8ac8f6a"
format_stamp: "Formatted at 2026-02-03 21:06:55 on dist-test-slave-j889"
server_key: "c64092814f560b17d4321dbe1b63552c"
server_key_iv: "6490611877968730d755417daa22f138"
server_key_version: "encryptionkey@0"
I20260203 21:06:55.660449 4023 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "4c580447d1e44ae1b5aba3fbd8ac8f6a"
format_stamp: "Formatted at 2026-02-03 21:06:55 on dist-test-slave-j889"
server_key: "c64092814f560b17d4321dbe1b63552c"
server_key_iv: "6490611877968730d755417daa22f138"
server_key_version: "encryptionkey@0"
I20260203 21:06:55.663594 4023 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260203 21:06:55.665604 4038 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:55.666509 4023 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260203 21:06:55.666621 4023 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "4c580447d1e44ae1b5aba3fbd8ac8f6a"
format_stamp: "Formatted at 2026-02-03 21:06:55 on dist-test-slave-j889"
server_key: "c64092814f560b17d4321dbe1b63552c"
server_key_iv: "6490611877968730d755417daa22f138"
server_key_version: "encryptionkey@0"
I20260203 21:06:55.666700 4023 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:55.686743 4023 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:55.689330 4023 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:55.689513 4023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:55.696149 4023 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:44955
I20260203 21:06:55.696158 4090 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:44955 every 8 connection(s)
I20260203 21:06:55.697177 4023 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:06:55.699963 4091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:55.700241 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 4023
I20260203 21:06:55.700359 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:06:55.700572 410 external_mini_cluster.cc:1468] Setting key ec6ab8ab657c213dfe18379431497f06
I20260203 21:06:55.706784 4091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a: Bootstrap starting.
Feb 03 21:06:55 dist-test-slave-j889 krb5kdc[4007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:55.709200 4091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:55.709893 4091 log.cc:826] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:55.711748 4091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a: No bootstrap required, opened a new log
I20260203 21:06:55.713086 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:55.701779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51096 (local address 127.0.102.190:44955)
0203 21:06:55.702162 (+ 383us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:55.702174 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:55.702214 (+ 40us) server_negotiation.cc:408] Connection header received
0203 21:06:55.703295 (+ 1081us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:55.703323 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:55.703708 (+ 385us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:55.704061 (+ 353us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:55.704834 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:55.705974 (+ 1140us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:55.706611 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:55.706874 (+ 263us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:55.709093 (+ 2219us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:55.709109 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:55.709120 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:55.709151 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:55.710734 (+ 1583us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:55.711172 (+ 438us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:55.711176 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:55.711180 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:55.711244 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:55.711469 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:55.711471 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:55.711472 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:55.711783 (+ 311us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:55.711950 (+ 167us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:55.712230 (+ 280us) server_negotiation.cc:300] Negotiation successful
0203 21:06:55.712462 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":94,"threads_started":1}
I20260203 21:06:55.714358 4091 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c580447d1e44ae1b5aba3fbd8ac8f6a" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 44955 } }
I20260203 21:06:55.714582 4091 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:55.714655 4091 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c580447d1e44ae1b5aba3fbd8ac8f6a, State: Initialized, Role: FOLLOWER
I20260203 21:06:55.715083 4091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c580447d1e44ae1b5aba3fbd8ac8f6a" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 44955 } }
I20260203 21:06:55.715232 4091 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:06:55.715292 4091 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:06:55.715363 4091 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:55.716110 4091 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c580447d1e44ae1b5aba3fbd8ac8f6a" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 44955 } }
I20260203 21:06:55.716408 4091 leader_election.cc:304] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c580447d1e44ae1b5aba3fbd8ac8f6a; no voters:
I20260203 21:06:55.716681 4091 leader_election.cc:290] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:06:55.717007 4096 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:55.717173 4096 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [term 1 LEADER]: Becoming Leader. State: Replica: 4c580447d1e44ae1b5aba3fbd8ac8f6a, State: Running, Role: LEADER
I20260203 21:06:55.717458 4096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c580447d1e44ae1b5aba3fbd8ac8f6a" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 44955 } }
I20260203 21:06:55.717803 4091 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:06:55.718879 4098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c580447d1e44ae1b5aba3fbd8ac8f6a. Latest consensus state: current_term: 1 leader_uuid: "4c580447d1e44ae1b5aba3fbd8ac8f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c580447d1e44ae1b5aba3fbd8ac8f6a" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 44955 } } }
I20260203 21:06:55.718981 4098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [sys.catalog]: This master's current role is: LEADER
I20260203 21:06:55.719213 4097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c580447d1e44ae1b5aba3fbd8ac8f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c580447d1e44ae1b5aba3fbd8ac8f6a" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 44955 } } }
I20260203 21:06:55.719301 4097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a [sys.catalog]: This master's current role is: LEADER
I20260203 21:06:55.719422 4105 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:06:55.722079 4105 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:06:55.726910 4105 catalog_manager.cc:1357] Generated new cluster ID: bdda5b03bdc0480f9d39e9f102405af3
I20260203 21:06:55.726981 4105 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:06:55.770049 4105 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:06:55.771103 4105 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:06:55.779536 4105 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 4c580447d1e44ae1b5aba3fbd8ac8f6a: Generated new TSK 0
I20260203 21:06:55.780238 4105 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:55.835811 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:44955
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:55.941380 4119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:55.941629 4119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:55.941716 4119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:55.945086 4119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:55.945159 4119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:55.945266 4119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:06:55.949611 4119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:44955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:55.950645 4119 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:55.951571 4119 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:55.958042 4124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:55.958189 4125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:55.958066 4127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:55.958326 4119 server_base.cc:1060] running on GCE node
I20260203 21:06:55.958669 4119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:55.959317 4119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:55.960485 4119 hybrid_clock.cc:648] HybridClock initialized: now 1770152815960463 us; error 36 us; skew 500 ppm
Feb 03 21:06:55 dist-test-slave-j889 krb5kdc[4007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:55.962998 4119 init.cc:377] Logged in from keytab as kudu/127.0.102.129@KRBTEST.COM (short username kudu)
I20260203 21:06:55.963994 4119 webserver.cc:492] Webserver started at http://127.0.102.129:39879/ using document root <none> and password file <none>
I20260203 21:06:55.964541 4119 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:55.964592 4119 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:55.964794 4119 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:55.966421 4119 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "ab2c205d1f034877b04436d650b18d06"
format_stamp: "Formatted at 2026-02-03 21:06:55 on dist-test-slave-j889"
server_key: "01539db8d5d9236cd2e0ae58675c8146"
server_key_iv: "da8e13deba14d3db31c78b0756a04cd3"
server_key_version: "encryptionkey@0"
I20260203 21:06:55.966853 4119 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "ab2c205d1f034877b04436d650b18d06"
format_stamp: "Formatted at 2026-02-03 21:06:55 on dist-test-slave-j889"
server_key: "01539db8d5d9236cd2e0ae58675c8146"
server_key_iv: "da8e13deba14d3db31c78b0756a04cd3"
server_key_version: "encryptionkey@0"
I20260203 21:06:55.970111 4119 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260203 21:06:55.972335 4134 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:55.973364 4119 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:06:55.973467 4119 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "ab2c205d1f034877b04436d650b18d06"
format_stamp: "Formatted at 2026-02-03 21:06:55 on dist-test-slave-j889"
server_key: "01539db8d5d9236cd2e0ae58675c8146"
server_key_iv: "da8e13deba14d3db31c78b0756a04cd3"
server_key_version: "encryptionkey@0"
I20260203 21:06:55.973554 4119 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:56.000900 4119 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:56.006448 4119 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:56.006626 4119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:56.007160 4119 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:56.008036 4119 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:56.008081 4119 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:56.008122 4119 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:56.008152 4119 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:56.016927 4119 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:36013
I20260203 21:06:56.016940 4247 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:36013 every 8 connection(s)
I20260203 21:06:56.017869 4119 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:06:56.021215 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 4119
I20260203 21:06:56.021327 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:06:56.021592 410 external_mini_cluster.cc:1468] Setting key 2b79b792fff30946f8ca84724d76ab6c
Feb 03 21:06:56 dist-test-slave-j889 krb5kdc[4007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.102.130@KRBTEST.COM; defaulting to no policy
I20260203 21:06:56.029534 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:56.019601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:58319 (local address 127.0.102.190:44955)
0203 21:06:56.019748 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:56.019752 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:56.020405 (+ 653us) server_negotiation.cc:408] Connection header received
0203 21:06:56.021256 (+ 851us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:56.021260 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:56.021314 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:56.021388 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:56.023239 (+ 1851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:56.023735 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:56.024317 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:56.024500 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:56.026525 (+ 2025us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:56.026541 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:56.026544 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:56.026566 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:56.027816 (+ 1250us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:56.028238 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:56.028241 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:56.028243 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:56.028283 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:56.028535 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:56.028538 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:56.028540 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:56.028723 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:56.028807 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:56.029250 (+ 443us) server_negotiation.cc:300] Negotiation successful
0203 21:06:56.029380 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260203 21:06:56.030236 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:56.019870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:44955 (local address 127.0.102.129:58319)
0203 21:06:56.020269 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:56.020304 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:56.021078 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:56.021549 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:56.021556 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:56.021987 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:56.023104 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:56.023119 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:56.023838 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:56.023841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:56.024217 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:56.024224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:56.024429 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:56.025005 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:56.025029 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:56.026393 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:56.027921 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:56.027926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:56.027937 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:56.028151 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:56.028355 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:56.028358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:56.028360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:56.028455 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:56.028817 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:56.028824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:56.029032 (+ 208us) client_negotiation.cc:769] Sending connection context
0203 21:06:56.029206 (+ 174us) client_negotiation.cc:240] Negotiation successful
0203 21:06:56.029465 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.130@KRBTEST.COM" created.
I20260203 21:06:56.031375 4248 heartbeater.cc:344] Connected to a master server at 127.0.102.190:44955
I20260203 21:06:56.031658 4248 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:56.032253 4248 heartbeater.cc:507] Master 127.0.102.190:44955 requested a full tablet report, sending...
I20260203 21:06:56.033782 4055 ts_manager.cc:194] Registered new tserver with Master: ab2c205d1f034877b04436d650b18d06 (127.0.102.129:36013)
I20260203 21:06:56.034914 4055 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:58319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:56.063402 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:44955
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:56.167213 4255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:56.167443 4255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:56.167505 4255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:56.170775 4255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:56.170837 4255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:56.170931 4255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:06:56.175006 4255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:44955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:56.175966 4255 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:56.176779 4255 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:56.183143 4261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:56.183238 4255 server_base.cc:1060] running on GCE node
W20260203 21:06:56.183058 4260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:56.183063 4263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:56.183686 4255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:56.184187 4255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:56.185338 4255 hybrid_clock.cc:648] HybridClock initialized: now 1770152816185321 us; error 28 us; skew 500 ppm
Feb 03 21:06:56 dist-test-slave-j889 krb5kdc[4007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:56.187690 4255 init.cc:377] Logged in from keytab as kudu/127.0.102.130@KRBTEST.COM (short username kudu)
I20260203 21:06:56.188616 4255 webserver.cc:492] Webserver started at http://127.0.102.130:45729/ using document root <none> and password file <none>
I20260203 21:06:56.189147 4255 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:56.189196 4255 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:56.189385 4255 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:56.190984 4255 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "104719bdaeae44a8802901b66502e69b"
format_stamp: "Formatted at 2026-02-03 21:06:56 on dist-test-slave-j889"
server_key: "9f0886404869c6982c0493a1323b298c"
server_key_iv: "775550ba3f8c03d224933bf49f3d0af8"
server_key_version: "encryptionkey@0"
I20260203 21:06:56.191422 4255 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "104719bdaeae44a8802901b66502e69b"
format_stamp: "Formatted at 2026-02-03 21:06:56 on dist-test-slave-j889"
server_key: "9f0886404869c6982c0493a1323b298c"
server_key_iv: "775550ba3f8c03d224933bf49f3d0af8"
server_key_version: "encryptionkey@0"
I20260203 21:06:56.194620 4255 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260203 21:06:56.196769 4270 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:56.197717 4255 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:06:56.197813 4255 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "104719bdaeae44a8802901b66502e69b"
format_stamp: "Formatted at 2026-02-03 21:06:56 on dist-test-slave-j889"
server_key: "9f0886404869c6982c0493a1323b298c"
server_key_iv: "775550ba3f8c03d224933bf49f3d0af8"
server_key_version: "encryptionkey@0"
I20260203 21:06:56.197890 4255 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:56.208864 4255 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:56.211448 4255 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:56.211607 4255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:56.212100 4255 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:56.212966 4255 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:56.213014 4255 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:56.213064 4255 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:56.213092 4255 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:56.221835 4255 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:36375
I20260203 21:06:56.221868 4383 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:36375 every 8 connection(s)
I20260203 21:06:56.222798 4255 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:06:56.228458 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 4255
I20260203 21:06:56.228571 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:06:56.228798 410 external_mini_cluster.cc:1468] Setting key b522ac6a6243ecb2062eb98b181103a6
Feb 03 21:06:56 dist-test-slave-j889 krb5kdc[4007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:56.234722 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:56.224434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:53121 (local address 127.0.102.190:44955)
0203 21:06:56.224555 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:56.224558 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:56.225286 (+ 728us) server_negotiation.cc:408] Connection header received
0203 21:06:56.226099 (+ 813us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:56.226102 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:56.226159 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:56.226228 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:56.227830 (+ 1602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:56.228376 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:56.229133 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:56.229305 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:56.231595 (+ 2290us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:56.231611 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:56.231613 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:56.231637 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:56.232915 (+ 1278us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:56.233396 (+ 481us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:56.233399 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:56.233400 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:56.233448 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:56.233706 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:56.233709 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:56.233712 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:56.233851 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:56.233932 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:56.234450 (+ 518us) server_negotiation.cc:300] Negotiation successful
0203 21:06:56.234569 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260203 21:06:56.235292 4386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:56.224760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:44955 (local address 127.0.102.130:53121)
0203 21:06:56.225154 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:56.225189 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:56.225911 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:56.226391 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:56.226399 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:56.226780 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:56.227686 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:56.227700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:56.228516 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:56.228519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:56.229007 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:56.229016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:56.229186 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:56.229841 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:56.229877 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:56.231450 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:56.233020 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:56.233025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:56.233045 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:56.233303 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:56.233527 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:56.233530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:56.233531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:56.233621 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:56.233955 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:56.233962 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:56.234181 (+ 219us) client_negotiation.cc:769] Sending connection context
0203 21:06:56.234379 (+ 198us) client_negotiation.cc:240] Negotiation successful
0203 21:06:56.234626 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":97,"threads_started":1}
I20260203 21:06:56.236420 4384 heartbeater.cc:344] Connected to a master server at 127.0.102.190:44955
I20260203 21:06:56.236667 4384 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.0.102.131@KRBTEST.COM; defaulting to no policy
I20260203 21:06:56.237156 4384 heartbeater.cc:507] Master 127.0.102.190:44955 requested a full tablet report, sending...
I20260203 21:06:56.238125 4055 ts_manager.cc:194] Registered new tserver with Master: 104719bdaeae44a8802901b66502e69b (127.0.102.130:36375)
I20260203 21:06:56.238622 4055 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.130@KRBTEST.COM'} at 127.0.102.130:53121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:56.269116 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:44955
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:56.371127 4391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:56.371332 4391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:56.371404 4391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:56.374891 4391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:56.374946 4391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:56.375041 4391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:06:56.379150 4391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:44955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:56.380159 4391 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:56.381021 4391 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:56.387372 4399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:56.387420 4396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:56.387498 4397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:56.387674 4391 server_base.cc:1060] running on GCE node
I20260203 21:06:56.387996 4391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:56.388514 4391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:56.389668 4391 hybrid_clock.cc:648] HybridClock initialized: now 1770152816389669 us; error 51 us; skew 500 ppm
Feb 03 21:06:56 dist-test-slave-j889 krb5kdc[4007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:56.392055 4391 init.cc:377] Logged in from keytab as kudu/127.0.102.131@KRBTEST.COM (short username kudu)
I20260203 21:06:56.393095 4391 webserver.cc:492] Webserver started at http://127.0.102.131:40619/ using document root <none> and password file <none>
I20260203 21:06:56.393631 4391 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:56.393683 4391 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:56.393852 4391 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:56.395520 4391 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "f562617f9422499da78f81e0dbe0581c"
format_stamp: "Formatted at 2026-02-03 21:06:56 on dist-test-slave-j889"
server_key: "0e938a0ac22d09fa7f4c940f83ff1bc5"
server_key_iv: "09e7ee3a0e1c868202ff788663532f47"
server_key_version: "encryptionkey@0"
I20260203 21:06:56.395962 4391 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "f562617f9422499da78f81e0dbe0581c"
format_stamp: "Formatted at 2026-02-03 21:06:56 on dist-test-slave-j889"
server_key: "0e938a0ac22d09fa7f4c940f83ff1bc5"
server_key_iv: "09e7ee3a0e1c868202ff788663532f47"
server_key_version: "encryptionkey@0"
I20260203 21:06:56.399082 4391 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260203 21:06:56.401258 4406 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:56.402324 4391 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:06:56.402448 4391 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "f562617f9422499da78f81e0dbe0581c"
format_stamp: "Formatted at 2026-02-03 21:06:56 on dist-test-slave-j889"
server_key: "0e938a0ac22d09fa7f4c940f83ff1bc5"
server_key_iv: "09e7ee3a0e1c868202ff788663532f47"
server_key_version: "encryptionkey@0"
I20260203 21:06:56.402534 4391 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:56.422597 4391 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:56.425400 4391 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:56.425585 4391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:56.426120 4391 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:56.426970 4391 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:56.427017 4391 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:56.427068 4391 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:56.427098 4391 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:56.436648 4391 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:34689
I20260203 21:06:56.436646 4519 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:34689 every 8 connection(s)
I20260203 21:06:56.437625 4391 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
I20260203 21:06:56.443724 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 4391
I20260203 21:06:56.443830 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:06:56.444036 410 external_mini_cluster.cc:1468] Setting key 24b9a020e80723d05566be25a9d531ef
Feb 03 21:06:56 dist-test-slave-j889 krb5kdc[4007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:56.449069 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:56.439567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:51519 (local address 127.0.102.190:44955)
0203 21:06:56.439726 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:56.439731 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:56.440371 (+ 640us) server_negotiation.cc:408] Connection header received
0203 21:06:56.441279 (+ 908us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:56.441282 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:56.441340 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:56.441411 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:56.442999 (+ 1588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:56.443501 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:56.444178 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:56.444359 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:56.446385 (+ 2026us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:56.446399 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:56.446401 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:56.446426 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:56.447517 (+ 1091us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:56.447922 (+ 405us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:56.447924 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:56.447926 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:56.447962 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:56.448177 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:56.448179 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:56.448181 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:56.448315 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:56.448381 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:56.448860 (+ 479us) server_negotiation.cc:300] Negotiation successful
0203 21:06:56.448954 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260203 21:06:56.449988 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:56.439832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:44955 (local address 127.0.102.131:51519)
0203 21:06:56.440241 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:56.440274 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:56.441083 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:56.441562 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:56.441589 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:56.441949 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:56.442853 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:56.442866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:56.443634 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:56.443638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:56.444073 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:56.444081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:56.444261 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:56.444827 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:56.444850 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:56.446263 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:56.447610 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:56.447615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:56.447623 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:56.447848 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:56.448023 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:56.448025 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:56.448026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:56.448114 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:56.448383 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:56.448387 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:56.448593 (+ 206us) client_negotiation.cc:769] Sending connection context
0203 21:06:56.448822 (+ 229us) client_negotiation.cc:240] Negotiation successful
0203 21:06:56.449023 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":99,"threads_started":1}
I20260203 21:06:56.450949 4520 heartbeater.cc:344] Connected to a master server at 127.0.102.190:44955
I20260203 21:06:56.451156 4520 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:56.451633 4520 heartbeater.cc:507] Master 127.0.102.190:44955 requested a full tablet report, sending...
I20260203 21:06:56.452505 4055 ts_manager.cc:194] Registered new tserver with Master: f562617f9422499da78f81e0dbe0581c (127.0.102.131:34689)
I20260203 21:06:56.453042 4055 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.131@KRBTEST.COM'} at 127.0.102.131:51519
I20260203 21:06:56.457594 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 03 21:06:56 dist-test-slave-j889 krb5kdc[4007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152816, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Feb 03 21:06:56 dist-test-slave-j889 krb5kdc[4007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152816, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:56.479542 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:56.472392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.0.102.190:44955)
0203 21:06:56.472555 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:56.472560 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:56.472653 (+ 93us) server_negotiation.cc:408] Connection header received
0203 21:06:56.472807 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:56.472809 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:56.472857 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:56.472924 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:56.473755 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:56.474247 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:56.474902 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:56.475086 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:56.477051 (+ 1965us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:56.477071 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:56.477074 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:56.477101 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:56.478424 (+ 1323us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:56.478732 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:56.478735 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:56.478736 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:56.478775 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:56.478913 (+ 138us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:56.478915 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:56.478916 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:56.479069 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:56.479159 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:56.479294 (+ 135us) server_negotiation.cc:300] Negotiation successful
0203 21:06:56.479383 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260203 21:06:56.479899 4055 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51112
I20260203 21:06:56.480659 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 4119
I20260203 21:06:56.486173 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 4255
I20260203 21:06:56.490957 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 4391
I20260203 21:06:56.495513 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 4023
2026-02-03T21:06:56Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2917 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:06:56 dist-test-slave-j889 krb5kdc[4532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:06:56 dist-test-slave-j889 krb5kdc[4532](info): set up 2 sockets
Feb 03 21:06:56 dist-test-slave-j889 krb5kdc[4532](info): commencing operation
krb5kdc: starting...
W20260203 21:06:58.341535 410 mini_kdc.cc:121] Time spent starting KDC: real 1.826s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:06:58 dist-test-slave-j889 krb5kdc[4532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:06:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:06:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:58.484012 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:45405
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:45405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:58.590188 4548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:58.590427 4548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:58.590487 4548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:58.593839 4548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:06:58.593909 4548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:58.593931 4548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:58.593950 4548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:06:58.593966 4548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:06:58.598311 4548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:45405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:45405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:58.599385 4548 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:58.600186 4548 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:58.605954 4553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:58.605907 4554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:58.606045 4548 server_base.cc:1060] running on GCE node
W20260203 21:06:58.606161 4556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:58.606616 4548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:58.607481 4548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:58.608661 4548 hybrid_clock.cc:648] HybridClock initialized: now 1770152818608635 us; error 47 us; skew 500 ppm
Feb 03 21:06:58 dist-test-slave-j889 krb5kdc[4532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:58.611121 4548 init.cc:377] Logged in from keytab as kudu/127.0.102.190@KRBTEST.COM (short username kudu)
I20260203 21:06:58.612128 4548 webserver.cc:492] Webserver started at http://127.0.102.190:41297/ using document root <none> and password file <none>
I20260203 21:06:58.612679 4548 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:58.612733 4548 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:58.612907 4548 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:58.614526 4548 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "b89586b3247d4c8395b2ac12cacd6bbf"
format_stamp: "Formatted at 2026-02-03 21:06:58 on dist-test-slave-j889"
server_key: "48344c1ea4856fdeb93c6103fd9b6281"
server_key_iv: "45fc0c0dfcd52d2f363795f80800c49c"
server_key_version: "encryptionkey@0"
I20260203 21:06:58.614981 4548 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "b89586b3247d4c8395b2ac12cacd6bbf"
format_stamp: "Formatted at 2026-02-03 21:06:58 on dist-test-slave-j889"
server_key: "48344c1ea4856fdeb93c6103fd9b6281"
server_key_iv: "45fc0c0dfcd52d2f363795f80800c49c"
server_key_version: "encryptionkey@0"
I20260203 21:06:58.618278 4548 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260203 21:06:58.620398 4563 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:58.621487 4548 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:06:58.621606 4548 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "b89586b3247d4c8395b2ac12cacd6bbf"
format_stamp: "Formatted at 2026-02-03 21:06:58 on dist-test-slave-j889"
server_key: "48344c1ea4856fdeb93c6103fd9b6281"
server_key_iv: "45fc0c0dfcd52d2f363795f80800c49c"
server_key_version: "encryptionkey@0"
I20260203 21:06:58.621734 4548 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:58.650432 4548 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:58.653235 4548 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:58.653402 4548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:58.661018 4615 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:45405 every 8 connection(s)
I20260203 21:06:58.661028 4548 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:45405
I20260203 21:06:58.662220 4548 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:06:58.665122 4616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:06:58.669864 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 4548
I20260203 21:06:58.669998 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:06:58.670234 410 external_mini_cluster.cc:1468] Setting key 621e66348eaf45f493164b29d7b148ab
I20260203 21:06:58.671090 4616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf: Bootstrap starting.
I20260203 21:06:58.673499 4616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf: Neither blocks nor log segments found. Creating new log.
I20260203 21:06:58.674340 4616 log.cc:826] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf: Log is configured to *not* fsync() on all Append() calls
I20260203 21:06:58.676331 4616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf: No bootstrap required, opened a new log
Feb 03 21:06:58 dist-test-slave-j889 krb5kdc[4532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:58.679477 4616 raft_consensus.cc:359] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89586b3247d4c8395b2ac12cacd6bbf" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 45405 } }
I20260203 21:06:58.679700 4616 raft_consensus.cc:385] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:06:58.679790 4616 raft_consensus.cc:740] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89586b3247d4c8395b2ac12cacd6bbf, State: Initialized, Role: FOLLOWER
I20260203 21:06:58.680222 4616 consensus_queue.cc:260] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89586b3247d4c8395b2ac12cacd6bbf" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 45405 } }
I20260203 21:06:58.680346 4616 raft_consensus.cc:399] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:06:58.680401 4616 raft_consensus.cc:493] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:06:58.680486 4616 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:06:58.681512 4616 raft_consensus.cc:515] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89586b3247d4c8395b2ac12cacd6bbf" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 45405 } }
I20260203 21:06:58.681895 4616 leader_election.cc:304] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b89586b3247d4c8395b2ac12cacd6bbf; no voters:
I20260203 21:06:58.682197 4616 leader_election.cc:290] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:06:58.682322 4621 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:06:58.682605 4621 raft_consensus.cc:697] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [term 1 LEADER]: Becoming Leader. State: Replica: b89586b3247d4c8395b2ac12cacd6bbf, State: Running, Role: LEADER
I20260203 21:06:58.683024 4621 consensus_queue.cc:237] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89586b3247d4c8395b2ac12cacd6bbf" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 45405 } }
I20260203 21:06:58.683413 4616 sys_catalog.cc:565] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:06:58.684252 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:58.671543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48790 (local address 127.0.102.190:45405)
0203 21:06:58.671986 (+ 443us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:58.671996 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:58.672032 (+ 36us) server_negotiation.cc:408] Connection header received
0203 21:06:58.672576 (+ 544us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:58.672593 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:58.672956 (+ 363us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:58.673246 (+ 290us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:58.674248 (+ 1002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:58.675085 (+ 837us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:58.675949 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:58.676187 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:58.678813 (+ 2626us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:58.678835 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:58.678846 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:58.678887 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:58.680914 (+ 2027us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:58.681425 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:58.681431 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:58.681435 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:58.681522 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:58.681806 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:58.681808 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:58.681810 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:58.682230 (+ 420us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:58.683013 (+ 783us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:58.683203 (+ 190us) server_negotiation.cc:300] Negotiation successful
0203 21:06:58.683482 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":134,"threads_started":1}
I20260203 21:06:58.684446 4623 sys_catalog.cc:455] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b89586b3247d4c8395b2ac12cacd6bbf. Latest consensus state: current_term: 1 leader_uuid: "b89586b3247d4c8395b2ac12cacd6bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89586b3247d4c8395b2ac12cacd6bbf" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 45405 } } }
I20260203 21:06:58.684547 4623 sys_catalog.cc:458] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [sys.catalog]: This master's current role is: LEADER
I20260203 21:06:58.685189 4622 sys_catalog.cc:455] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b89586b3247d4c8395b2ac12cacd6bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89586b3247d4c8395b2ac12cacd6bbf" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 45405 } } }
I20260203 21:06:58.685432 4622 sys_catalog.cc:458] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf [sys.catalog]: This master's current role is: LEADER
I20260203 21:06:58.686605 4630 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:06:58.689340 4630 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:06:58.694633 4630 catalog_manager.cc:1357] Generated new cluster ID: a8d8900072b34b7dbfbcc7b77caa4707
I20260203 21:06:58.694746 4630 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:06:58.704357 4630 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:06:58.705731 4630 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:06:58.718147 4630 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b89586b3247d4c8395b2ac12cacd6bbf: Generated new TSK 0
I20260203 21:06:58.719110 4630 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:58.785256 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:45405
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:58.900995 4644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:58.901278 4644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:58.901361 4644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:58.905504 4644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:58.905591 4644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:58.905694 4644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:06:58.910466 4644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:45405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:58.911689 4644 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:58.912717 4644 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:58.920795 4650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:58.920819 4649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:58.920998 4644 server_base.cc:1060] running on GCE node
W20260203 21:06:58.921043 4652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:58.921447 4644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:58.922083 4644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:58.923259 4644 hybrid_clock.cc:648] HybridClock initialized: now 1770152818923207 us; error 68 us; skew 500 ppm
Feb 03 21:06:58 dist-test-slave-j889 krb5kdc[4532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:58.926095 4644 init.cc:377] Logged in from keytab as kudu/127.0.102.129@KRBTEST.COM (short username kudu)
I20260203 21:06:58.927242 4644 webserver.cc:492] Webserver started at http://127.0.102.129:40635/ using document root <none> and password file <none>
I20260203 21:06:58.927785 4644 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:58.927842 4644 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:58.928018 4644 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:58.929821 4644 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "fbd124dca72d4d79973d3180f9f92eeb"
format_stamp: "Formatted at 2026-02-03 21:06:58 on dist-test-slave-j889"
server_key: "21293b4b2240e0062be27b8446d4186f"
server_key_iv: "6a2a94a2b99be9a6b94c25721855da3b"
server_key_version: "encryptionkey@0"
I20260203 21:06:58.930322 4644 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "fbd124dca72d4d79973d3180f9f92eeb"
format_stamp: "Formatted at 2026-02-03 21:06:58 on dist-test-slave-j889"
server_key: "21293b4b2240e0062be27b8446d4186f"
server_key_iv: "6a2a94a2b99be9a6b94c25721855da3b"
server_key_version: "encryptionkey@0"
I20260203 21:06:58.934029 4644 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260203 21:06:58.936363 4659 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:58.937602 4644 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:06:58.937726 4644 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "fbd124dca72d4d79973d3180f9f92eeb"
format_stamp: "Formatted at 2026-02-03 21:06:58 on dist-test-slave-j889"
server_key: "21293b4b2240e0062be27b8446d4186f"
server_key_iv: "6a2a94a2b99be9a6b94c25721855da3b"
server_key_version: "encryptionkey@0"
I20260203 21:06:58.937855 4644 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:58.978195 4644 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:58.981559 4644 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:58.981761 4644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:58.982381 4644 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:58.983314 4644 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:58.983363 4644 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:58.983404 4644 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:58.983424 4644 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:58.994270 4644 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:43171
I20260203 21:06:58.994287 4772 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:43171 every 8 connection(s)
I20260203 21:06:58.995297 4644 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:06:59.002460 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 4644
I20260203 21:06:59.002604 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:06:59.002842 410 external_mini_cluster.cc:1468] Setting key 0b031161086aca2c01c851ae6cfe3245
Feb 03 21:06:58 dist-test-slave-j889 krb5kdc[4532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:59.009584 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:58.997163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:56885 (local address 127.0.102.190:45405)
0203 21:06:58.997337 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:58.997343 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:58.998060 (+ 717us) server_negotiation.cc:408] Connection header received
0203 21:06:58.998919 (+ 859us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:58.998922 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:58.998973 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:58.999058 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:59.001046 (+ 1988us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:59.001561 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:59.002314 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:59.002504 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:59.005264 (+ 2760us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:59.005288 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:59.005291 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:59.005322 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:59.007064 (+ 1742us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:59.007707 (+ 643us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:59.007711 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:59.007713 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:59.007797 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:59.008208 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:59.008212 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:59.008213 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:59.008393 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:59.008485 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:59.009285 (+ 800us) server_negotiation.cc:300] Negotiation successful
0203 21:06:59.009441 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260203 21:06:59.010448 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:58.997429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:45405 (local address 127.0.102.129:56885)
0203 21:06:58.997908 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:58.997950 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:58.998711 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:58.999343 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:58.999356 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:58.999852 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:59.000852 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:59.000869 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:59.001702 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:59.001706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:59.002159 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:59.002170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:59.002436 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:59.003067 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:59.003109 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:59.005059 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:59.007209 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:59.007216 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:59.007230 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:59.007576 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:59.007914 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:59.007918 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:59.007920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:59.008071 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:59.008512 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:59.008520 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:59.008918 (+ 398us) client_negotiation.cc:769] Sending connection context
0203 21:06:59.009203 (+ 285us) client_negotiation.cc:240] Negotiation successful
0203 21:06:59.009533 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
I20260203 21:06:59.011821 4773 heartbeater.cc:344] Connected to a master server at 127.0.102.190:45405
I20260203 21:06:59.012111 4773 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.0.102.130@KRBTEST.COM; defaulting to no policy
I20260203 21:06:59.012825 4773 heartbeater.cc:507] Master 127.0.102.190:45405 requested a full tablet report, sending...
I20260203 21:06:59.014479 4576 ts_manager.cc:194] Registered new tserver with Master: fbd124dca72d4d79973d3180f9f92eeb (127.0.102.129:43171)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.130@KRBTEST.COM" created.
I20260203 21:06:59.015846 4576 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:56885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:59.062831 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:45405
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:59.178032 4780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:59.178280 4780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:59.178341 4780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:59.181867 4780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:59.181965 4780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:59.182050 4780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:06:59.187079 4780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:45405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:59.188205 4780 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:59.189136 4780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:59.197026 4788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:59.197026 4785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:59.197144 4780 server_base.cc:1060] running on GCE node
W20260203 21:06:59.197031 4786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:59.197674 4780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:59.198285 4780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:59.199445 4780 hybrid_clock.cc:648] HybridClock initialized: now 1770152819199422 us; error 38 us; skew 500 ppm
Feb 03 21:06:59 dist-test-slave-j889 krb5kdc[4532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:59.202289 4780 init.cc:377] Logged in from keytab as kudu/127.0.102.130@KRBTEST.COM (short username kudu)
I20260203 21:06:59.203398 4780 webserver.cc:492] Webserver started at http://127.0.102.130:38771/ using document root <none> and password file <none>
I20260203 21:06:59.203953 4780 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:59.204005 4780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:59.204185 4780 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:59.205986 4780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "51a0c71538df4f119468ad8a29acda1b"
format_stamp: "Formatted at 2026-02-03 21:06:59 on dist-test-slave-j889"
server_key: "e224c06558f81217769b97dcf1f05a04"
server_key_iv: "69f158d0b60714893f7fe2157ebe0731"
server_key_version: "encryptionkey@0"
I20260203 21:06:59.206478 4780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "51a0c71538df4f119468ad8a29acda1b"
format_stamp: "Formatted at 2026-02-03 21:06:59 on dist-test-slave-j889"
server_key: "e224c06558f81217769b97dcf1f05a04"
server_key_iv: "69f158d0b60714893f7fe2157ebe0731"
server_key_version: "encryptionkey@0"
I20260203 21:06:59.210367 4780 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260203 21:06:59.213210 4795 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:59.214527 4780 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260203 21:06:59.214674 4780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "51a0c71538df4f119468ad8a29acda1b"
format_stamp: "Formatted at 2026-02-03 21:06:59 on dist-test-slave-j889"
server_key: "e224c06558f81217769b97dcf1f05a04"
server_key_iv: "69f158d0b60714893f7fe2157ebe0731"
server_key_version: "encryptionkey@0"
I20260203 21:06:59.214787 4780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:59.233842 4780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:59.237264 4780 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:59.237470 4780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:59.238073 4780 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:59.239025 4780 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:59.239074 4780 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:59.239128 4780 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:59.239157 4780 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:59.250669 4780 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:43277
I20260203 21:06:59.250698 4908 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:43277 every 8 connection(s)
I20260203 21:06:59.251830 4780 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:06:59.259527 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 4780
I20260203 21:06:59.259646 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:06:59.259883 410 external_mini_cluster.cc:1468] Setting key c80eea4f72d2383d5cb1bdf6dbda702e
Feb 03 21:06:59 dist-test-slave-j889 krb5kdc[4532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:59.267220 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:59.254046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:43671 (local address 127.0.102.190:45405)
0203 21:06:59.254235 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:59.254240 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:59.254991 (+ 751us) server_negotiation.cc:408] Connection header received
0203 21:06:59.256032 (+ 1041us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:59.256035 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:59.256096 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:59.256168 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:59.258162 (+ 1994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:59.258700 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:59.259505 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:59.259703 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:59.262362 (+ 2659us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:59.262385 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:59.262388 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:59.262421 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:59.264241 (+ 1820us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:59.264987 (+ 746us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:59.264990 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:59.264992 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:59.265078 (+ 86us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:59.265536 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:59.265539 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:59.265541 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:59.265730 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:59.265821 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:59.266634 (+ 813us) server_negotiation.cc:300] Negotiation successful
0203 21:06:59.266961 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260203 21:06:59.267791 4911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:59.254321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:45405 (local address 127.0.102.130:43671)
0203 21:06:59.254843 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:59.254880 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:59.255656 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:59.256383 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:59.256393 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:59.256895 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:59.257864 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:59.257880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:59.258872 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:59.258876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:59.259356 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:59.259364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:59.259638 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:59.260288 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:59.260322 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:59.262163 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:59.264401 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:59.264414 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:59.264429 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:59.264837 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:59.265201 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:59.265204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:59.265206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:59.265366 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:59.265857 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:59.265866 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:59.266251 (+ 385us) client_negotiation.cc:769] Sending connection context
0203 21:06:59.266530 (+ 279us) client_negotiation.cc:240] Negotiation successful
0203 21:06:59.266873 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":116,"threads_started":1}
I20260203 21:06:59.269088 4909 heartbeater.cc:344] Connected to a master server at 127.0.102.190:45405
I20260203 21:06:59.269384 4909 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.0.102.131@KRBTEST.COM; defaulting to no policy
I20260203 21:06:59.270087 4909 heartbeater.cc:507] Master 127.0.102.190:45405 requested a full tablet report, sending...
I20260203 21:06:59.271276 4576 ts_manager.cc:194] Registered new tserver with Master: 51a0c71538df4f119468ad8a29acda1b (127.0.102.130:43277)
I20260203 21:06:59.271875 4576 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.130@KRBTEST.COM'} at 127.0.102.130:43671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:06:59.322723 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:45405
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:06:59.435317 4916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:06:59.435575 4916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:06:59.435635 4916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:06:59.439139 4916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:06:59.439222 4916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:06:59.439306 4916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:06:59.443966 4916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:45405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:06:59.445225 4916 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:06:59.446116 4916 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:06:59.453294 4921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:59.453979 4924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:06:59.454013 4922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:06:59.454149 4916 server_base.cc:1060] running on GCE node
I20260203 21:06:59.454640 4916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:06:59.455262 4916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:06:59.456436 4916 hybrid_clock.cc:648] HybridClock initialized: now 1770152819456408 us; error 42 us; skew 500 ppm
Feb 03 21:06:59 dist-test-slave-j889 krb5kdc[4532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:06:59.459419 4916 init.cc:377] Logged in from keytab as kudu/127.0.102.131@KRBTEST.COM (short username kudu)
I20260203 21:06:59.460875 4916 webserver.cc:492] Webserver started at http://127.0.102.131:39101/ using document root <none> and password file <none>
I20260203 21:06:59.461468 4916 fs_manager.cc:362] Metadata directory not provided
I20260203 21:06:59.461521 4916 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:06:59.461695 4916 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:06:59.463575 4916 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "c4c2aa8d19aa423ab63bf6fa951ec86c"
format_stamp: "Formatted at 2026-02-03 21:06:59 on dist-test-slave-j889"
server_key: "7f04d6dc0cdd9f78c2b5f5e00a74c950"
server_key_iv: "1a497a9d57a179d0c2d9822af22c9191"
server_key_version: "encryptionkey@0"
I20260203 21:06:59.464058 4916 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "c4c2aa8d19aa423ab63bf6fa951ec86c"
format_stamp: "Formatted at 2026-02-03 21:06:59 on dist-test-slave-j889"
server_key: "7f04d6dc0cdd9f78c2b5f5e00a74c950"
server_key_iv: "1a497a9d57a179d0c2d9822af22c9191"
server_key_version: "encryptionkey@0"
I20260203 21:06:59.467720 4916 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260203 21:06:59.470080 4931 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:59.471252 4916 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260203 21:06:59.471359 4916 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "c4c2aa8d19aa423ab63bf6fa951ec86c"
format_stamp: "Formatted at 2026-02-03 21:06:59 on dist-test-slave-j889"
server_key: "7f04d6dc0cdd9f78c2b5f5e00a74c950"
server_key_iv: "1a497a9d57a179d0c2d9822af22c9191"
server_key_version: "encryptionkey@0"
I20260203 21:06:59.471475 4916 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:06:59.492197 4916 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:06:59.495193 4916 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:06:59.495371 4916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:06:59.495944 4916 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:06:59.496891 4916 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:06:59.496944 4916 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:59.497004 4916 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:06:59.497032 4916 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:06:59.507613 4916 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:36385
I20260203 21:06:59.507743 5044 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:36385 every 8 connection(s)
I20260203 21:06:59.508692 4916 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
I20260203 21:06:59.509548 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 4916
I20260203 21:06:59.509665 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:06:59.509914 410 external_mini_cluster.cc:1468] Setting key 552efcf626f7b552e89fdfca205ee37a
Feb 03 21:06:59 dist-test-slave-j889 krb5kdc[4532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:59.522360 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:59.510741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:35373 (local address 127.0.102.190:45405)
0203 21:06:59.510886 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:59.510891 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:59.511771 (+ 880us) server_negotiation.cc:408] Connection header received
0203 21:06:59.512701 (+ 930us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:59.512705 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:59.512755 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:59.512827 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:59.514577 (+ 1750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:59.515124 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:59.515990 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:59.516162 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:59.518672 (+ 2510us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:59.518696 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:59.518698 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:59.518727 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:59.520277 (+ 1550us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:59.520847 (+ 570us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:59.520850 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:59.520851 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:59.520907 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:59.521202 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:59.521206 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:59.521208 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:59.521378 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:59.521471 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:59.522022 (+ 551us) server_negotiation.cc:300] Negotiation successful
0203 21:06:59.522158 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260203 21:06:59.523146 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:59.511102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:45405 (local address 127.0.102.131:35373)
0203 21:06:59.511624 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0203 21:06:59.511669 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0203 21:06:59.512465 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:06:59.512987 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:06:59.512996 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:06:59.513442 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:06:59.514403 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:59.514419 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:59.515374 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:06:59.515378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:06:59.515839 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:06:59.515851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:06:59.516086 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:59.516894 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:06:59.516930 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:06:59.518489 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:06:59.520427 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:59.520433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:59.520445 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:59.520745 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:59.520996 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:06:59.520999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:06:59.521002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:06:59.521112 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:06:59.521482 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:06:59.521490 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:06:59.521750 (+ 260us) client_negotiation.cc:769] Sending connection context
0203 21:06:59.521970 (+ 220us) client_negotiation.cc:240] Negotiation successful
0203 21:06:59.522276 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":128,"threads_started":1}
I20260203 21:06:59.524226 5045 heartbeater.cc:344] Connected to a master server at 127.0.102.190:45405
I20260203 21:06:59.524511 5045 heartbeater.cc:461] Registering TS with master...
I20260203 21:06:59.525110 5045 heartbeater.cc:507] Master 127.0.102.190:45405 requested a full tablet report, sending...
I20260203 21:06:59.526262 4576 ts_manager.cc:194] Registered new tserver with Master: c4c2aa8d19aa423ab63bf6fa951ec86c (127.0.102.131:36385)
I20260203 21:06:59.526823 4576 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.131@KRBTEST.COM'} at 127.0.102.131:35373
I20260203 21:06:59.536898 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 03 21:06:59 dist-test-slave-j889 krb5kdc[4532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 03 21:06:59 dist-test-slave-j889 krb5kdc[4532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.129@KRBTEST.COM
I20260203 21:06:59.561275 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:59.552649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.0.102.129:43171)
0203 21:06:59.552907 (+ 258us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:59.552913 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:59.552936 (+ 23us) server_negotiation.cc:408] Connection header received
0203 21:06:59.552988 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:59.552993 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:59.553165 (+ 172us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:59.553277 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:59.554113 (+ 836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:59.554891 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:59.555605 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:59.555832 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:59.557950 (+ 2118us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:59.557978 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:59.557990 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:59.558020 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:59.559597 (+ 1577us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:59.560001 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:59.560006 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:59.560007 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:59.560068 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:59.560286 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:59.560289 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:59.560291 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:59.560507 (+ 216us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:59.560736 (+ 229us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:59.560953 (+ 217us) server_negotiation.cc:300] Negotiation successful
0203 21:06:59.561105 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
Feb 03 21:06:59 dist-test-slave-j889 krb5kdc[4532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:06:59.571061 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:06:59.563915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.0.102.190:45405)
0203 21:06:59.564060 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0203 21:06:59.564064 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:06:59.564077 (+ 13us) server_negotiation.cc:408] Connection header received
0203 21:06:59.564118 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:06:59.564121 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:06:59.564162 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:06:59.564226 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:06:59.564984 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:59.565503 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:06:59.566154 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:06:59.566295 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:06:59.568317 (+ 2022us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:06:59.568340 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:06:59.568342 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:06:59.568372 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:06:59.569599 (+ 1227us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:59.569945 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:59.569948 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:59.569950 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:59.569995 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:06:59.570242 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:06:59.570244 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:06:59.570245 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:06:59.570416 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:06:59.570513 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0203 21:06:59.570773 (+ 260us) server_negotiation.cc:300] Negotiation successful
0203 21:06:59.570897 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260203 21:06:59.571491 4576 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48792
I20260203 21:06:59.572115 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 4644
I20260203 21:06:59.577417 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 4780
I20260203 21:06:59.582203 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 4916
I20260203 21:06:59.586925 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 4548
2026-02-03T21:06:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3091 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:06:59 dist-test-slave-j889 krb5kdc[5057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:06:59 dist-test-slave-j889 krb5kdc[5057](info): set up 2 sockets
Feb 03 21:06:59 dist-test-slave-j889 krb5kdc[5057](info): commencing operation
krb5kdc: starting...
W20260203 21:07:01.436883 410 mini_kdc.cc:121] Time spent starting KDC: real 1.829s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:07:01 dist-test-slave-j889 krb5kdc[5057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:07:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:07:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:07:01.559209 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:41819
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:41819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:01.660593 5073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:01.660871 5073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:01.660948 5073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:01.664239 5073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:07:01.664309 5073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:01.664340 5073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:01.664366 5073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:07:01.664402 5073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:07:01.668969 5073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:41819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:41819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:01.670192 5073 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:01.671080 5073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:01.676512 5081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:01.676560 5079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:01.676535 5078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:01.676844 5073 server_base.cc:1060] running on GCE node
I20260203 21:07:01.677318 5073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:01.678176 5073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:01.679319 5073 hybrid_clock.cc:648] HybridClock initialized: now 1770152821679300 us; error 34 us; skew 500 ppm
Feb 03 21:07:01 dist-test-slave-j889 krb5kdc[5057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:01.681669 5073 init.cc:377] Logged in from keytab as kudu/127.0.102.190@KRBTEST.COM (short username kudu)
I20260203 21:07:01.682337 5073 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:01.682394 5073 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:01.682566 5073 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:01.684242 5073 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "d35980e2cb91482ba888b8895d52ab7f"
format_stamp: "Formatted at 2026-02-03 21:07:01 on dist-test-slave-j889"
server_key: "17a229cc3bda865c89fe5391bb4bc41a"
server_key_iv: "179f6579a73314fe2d30c78c52a9157b"
server_key_version: "encryptionkey@0"
I20260203 21:07:01.684728 5073 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "d35980e2cb91482ba888b8895d52ab7f"
format_stamp: "Formatted at 2026-02-03 21:07:01 on dist-test-slave-j889"
server_key: "17a229cc3bda865c89fe5391bb4bc41a"
server_key_iv: "179f6579a73314fe2d30c78c52a9157b"
server_key_version: "encryptionkey@0"
I20260203 21:07:01.687907 5073 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260203 21:07:01.690055 5087 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:01.691018 5073 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260203 21:07:01.691121 5073 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "d35980e2cb91482ba888b8895d52ab7f"
format_stamp: "Formatted at 2026-02-03 21:07:01 on dist-test-slave-j889"
server_key: "17a229cc3bda865c89fe5391bb4bc41a"
server_key_iv: "179f6579a73314fe2d30c78c52a9157b"
server_key_version: "encryptionkey@0"
I20260203 21:07:01.691224 5073 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:01.718657 5073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:01.726142 5073 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:01.726332 5073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:01.733328 5073 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:41819
I20260203 21:07:01.733309 5139 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:41819 every 8 connection(s)
I20260203 21:07:01.734000 5073 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:07:01.736692 5140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:01.742317 5140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f: Bootstrap starting.
I20260203 21:07:01.744133 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 5073
I20260203 21:07:01.744247 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:07:01.744453 410 external_mini_cluster.cc:1468] Setting key 3d8803e611f0ac76a3d479bb9161ee30
I20260203 21:07:01.744679 5140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:01.745524 5140 log.cc:826] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:01.747489 5140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f: No bootstrap required, opened a new log
I20260203 21:07:01.750521 5140 raft_consensus.cc:359] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35980e2cb91482ba888b8895d52ab7f" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 41819 } }
I20260203 21:07:01.750759 5140 raft_consensus.cc:385] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:01.750825 5140 raft_consensus.cc:740] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35980e2cb91482ba888b8895d52ab7f, State: Initialized, Role: FOLLOWER
I20260203 21:07:01.751443 5140 consensus_queue.cc:260] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35980e2cb91482ba888b8895d52ab7f" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 41819 } }
I20260203 21:07:01.751581 5140 raft_consensus.cc:399] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:07:01.751646 5140 raft_consensus.cc:493] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:07:01.751732 5140 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [term 0 FOLLOWER]: Advancing to term 1
Feb 03 21:07:01 dist-test-slave-j889 krb5kdc[5057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:07:01.752732 5140 raft_consensus.cc:515] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35980e2cb91482ba888b8895d52ab7f" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 41819 } }
I20260203 21:07:01.753046 5140 leader_election.cc:304] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d35980e2cb91482ba888b8895d52ab7f; no voters:
I20260203 21:07:01.753316 5140 leader_election.cc:290] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:07:01.753418 5145 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:01.753643 5145 raft_consensus.cc:697] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [term 1 LEADER]: Becoming Leader. State: Replica: d35980e2cb91482ba888b8895d52ab7f, State: Running, Role: LEADER
I20260203 21:07:01.753965 5145 consensus_queue.cc:237] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35980e2cb91482ba888b8895d52ab7f" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 41819 } }
I20260203 21:07:01.754508 5140 sys_catalog.cc:565] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:07:01.755640 5146 sys_catalog.cc:455] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d35980e2cb91482ba888b8895d52ab7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35980e2cb91482ba888b8895d52ab7f" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 41819 } } }
I20260203 21:07:01.755649 5147 sys_catalog.cc:455] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d35980e2cb91482ba888b8895d52ab7f. Latest consensus state: current_term: 1 leader_uuid: "d35980e2cb91482ba888b8895d52ab7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35980e2cb91482ba888b8895d52ab7f" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 41819 } } }
I20260203 21:07:01.755749 5146 sys_catalog.cc:458] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:01.755761 5147 sys_catalog.cc:458] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:01.756078 5154 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:07:01.757560 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:01.745432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58024 (local address 127.0.102.190:41819)
0203 21:07:01.745824 (+ 392us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:01.745834 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:01.745867 (+ 33us) server_negotiation.cc:408] Connection header received
0203 21:07:01.746511 (+ 644us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:01.746532 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:01.746824 (+ 292us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:01.747098 (+ 274us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:01.748106 (+ 1008us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:01.748921 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:01.749763 (+ 842us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:01.749961 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:01.752343 (+ 2382us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:01.752365 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:01.752376 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:01.752409 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:01.754692 (+ 2283us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:01.755101 (+ 409us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:01.755106 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:01.755111 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:01.755211 (+ 100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:01.755489 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:01.755492 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:01.755494 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:01.755934 (+ 440us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:01.756117 (+ 183us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:01.756521 (+ 404us) server_negotiation.cc:300] Negotiation successful
0203 21:07:01.756814 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":103,"threads_started":1}
I20260203 21:07:01.759224 5154 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:07:01.764016 5154 catalog_manager.cc:1357] Generated new cluster ID: 52f99007277e41fca919d5985e2fd41a
I20260203 21:07:01.764089 5154 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:07:01.784566 5154 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:07:01.785516 5154 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:07:01.794860 5154 catalog_manager.cc:6027] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f: Generated new TSK 0
I20260203 21:07:01.795581 5154 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:07:01.847906 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:41819
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:01.952270 5168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:01.952498 5168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:01.952570 5168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:01.955869 5168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:01.955932 5168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:01.956033 5168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:07:01.960157 5168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:41819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:01.961199 5168 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:01.962018 5168 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:01.967952 5173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:01.967950 5174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:01.968122 5168 server_base.cc:1060] running on GCE node
W20260203 21:07:01.967970 5176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:01.968531 5168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:01.969048 5168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:01.970185 5168 hybrid_clock.cc:648] HybridClock initialized: now 1770152821970167 us; error 30 us; skew 500 ppm
Feb 03 21:07:01 dist-test-slave-j889 krb5kdc[5057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:01.972536 5168 init.cc:377] Logged in from keytab as kudu/127.0.102.129@KRBTEST.COM (short username kudu)
I20260203 21:07:01.973217 5168 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:01.973275 5168 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:01.973439 5168 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:01.975057 5168 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8"
format_stamp: "Formatted at 2026-02-03 21:07:01 on dist-test-slave-j889"
server_key: "b9eb5edbaf978eb7c876d06827438626"
server_key_iv: "24c9a09f92319c6b9ab2273529d63016"
server_key_version: "encryptionkey@0"
I20260203 21:07:01.975497 5168 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8"
format_stamp: "Formatted at 2026-02-03 21:07:01 on dist-test-slave-j889"
server_key: "b9eb5edbaf978eb7c876d06827438626"
server_key_iv: "24c9a09f92319c6b9ab2273529d63016"
server_key_version: "encryptionkey@0"
I20260203 21:07:01.978681 5168 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260203 21:07:01.980690 5182 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:01.981611 5168 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:01.981703 5168 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8"
format_stamp: "Formatted at 2026-02-03 21:07:01 on dist-test-slave-j889"
server_key: "b9eb5edbaf978eb7c876d06827438626"
server_key_iv: "24c9a09f92319c6b9ab2273529d63016"
server_key_version: "encryptionkey@0"
I20260203 21:07:01.981781 5168 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:02.022003 5168 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:02.024715 5168 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:02.024897 5168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:02.025440 5168 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:02.026307 5168 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:02.026355 5168 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:02.026407 5168 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:02.026434 5168 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:02.035142 5168 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:37815
I20260203 21:07:02.035167 5295 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:37815 every 8 connection(s)
I20260203 21:07:02.035686 5168 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
Feb 03 21:07:02 dist-test-slave-j889 krb5kdc[5057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:07:02.043558 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 5168
I20260203 21:07:02.043666 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:07:02.043887 410 external_mini_cluster.cc:1468] Setting key 93c174f185bda49de25cfa420d69ac0c
I20260203 21:07:02.047551 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.037272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:45805 (local address 127.0.102.190:41819)
0203 21:07:02.037404 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:02.037408 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:02.038052 (+ 644us) server_negotiation.cc:408] Connection header received
0203 21:07:02.038866 (+ 814us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:02.038870 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:02.038927 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:02.038997 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:02.040591 (+ 1594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.041119 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.041727 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.041898 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.043941 (+ 2043us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:02.043964 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:02.043967 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:02.043999 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:02.045717 (+ 1718us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:02.046152 (+ 435us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:02.046155 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:02.046157 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:02.046213 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:02.046440 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:02.046443 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:02.046444 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:02.046624 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:02.046716 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:02.047331 (+ 615us) server_negotiation.cc:300] Negotiation successful
0203 21:07:02.047433 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260203 21:07:02.048391 5298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.037551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:41819 (local address 127.0.102.129:45805)
0203 21:07:02.037928 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:02.037962 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:02.038682 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:02.039153 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:02.039161 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:02.039549 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:02.040444 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:02.040459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.041239 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.041242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:02.041625 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:02.041631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.041819 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.042362 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:02.042387 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:02.043803 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:02.045820 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:02.045824 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:02.045834 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:02.046070 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:02.046280 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:02.046283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:02.046284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:02.046373 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:02.046737 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:02.046744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:02.047052 (+ 308us) client_negotiation.cc:769] Sending connection context
0203 21:07:02.047301 (+ 249us) client_negotiation.cc:240] Negotiation successful
0203 21:07:02.047570 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":92,"threads_started":1}
I20260203 21:07:02.049517 5296 heartbeater.cc:344] Connected to a master server at 127.0.102.190:41819
I20260203 21:07:02.049746 5296 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:02.050199 5296 heartbeater.cc:507] Master 127.0.102.190:41819 requested a full tablet report, sending...
I20260203 21:07:02.051695 5104 ts_manager.cc:194] Registered new tserver with Master: 8f38e9e8daf14ef089ce0040f3ef5eb8 (127.0.102.129:37815)
WARNING: no policy specified for kudu/127.0.102.130@KRBTEST.COM; defaulting to no policy
I20260203 21:07:02.053208 5104 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:45805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:07:02.085561 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:41819
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:02.187295 5303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:02.187502 5303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:02.187572 5303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:02.190917 5303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:02.190972 5303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:02.191079 5303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:07:02.195199 5303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:41819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:02.196197 5303 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:02.197026 5303 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:02.203047 5308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:02.203065 5309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:02.203056 5311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:02.203253 5303 server_base.cc:1060] running on GCE node
I20260203 21:07:02.203687 5303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:02.204159 5303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:02.205294 5303 hybrid_clock.cc:648] HybridClock initialized: now 1770152822205280 us; error 26 us; skew 500 ppm
Feb 03 21:07:02 dist-test-slave-j889 krb5kdc[5057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:02.207656 5303 init.cc:377] Logged in from keytab as kudu/127.0.102.130@KRBTEST.COM (short username kudu)
I20260203 21:07:02.208293 5303 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:02.208351 5303 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:02.208520 5303 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:02.210162 5303 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "18ed75d8fa7d4431a3cb66847fba518f"
format_stamp: "Formatted at 2026-02-03 21:07:02 on dist-test-slave-j889"
server_key: "394ab3256e54b0c4f97029dfaedd9df1"
server_key_iv: "9754d1ecad5d0c9f314aedac173060b1"
server_key_version: "encryptionkey@0"
I20260203 21:07:02.210610 5303 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "18ed75d8fa7d4431a3cb66847fba518f"
format_stamp: "Formatted at 2026-02-03 21:07:02 on dist-test-slave-j889"
server_key: "394ab3256e54b0c4f97029dfaedd9df1"
server_key_iv: "9754d1ecad5d0c9f314aedac173060b1"
server_key_version: "encryptionkey@0"
I20260203 21:07:02.213783 5303 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260203 21:07:02.215868 5317 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:02.216849 5303 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:07:02.216946 5303 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "18ed75d8fa7d4431a3cb66847fba518f"
format_stamp: "Formatted at 2026-02-03 21:07:02 on dist-test-slave-j889"
server_key: "394ab3256e54b0c4f97029dfaedd9df1"
server_key_iv: "9754d1ecad5d0c9f314aedac173060b1"
server_key_version: "encryptionkey@0"
I20260203 21:07:02.217025 5303 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:02.243755 5303 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:02.246186 5303 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:02.246354 5303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:02.246912 5303 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:02.247745 5303 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:02.247792 5303 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:02.247844 5303 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:02.247871 5303 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:02.256531 5303 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:41061
I20260203 21:07:02.256557 5430 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:41061 every 8 connection(s)
I20260203 21:07:02.257146 5303 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:07:02.260226 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 5303
I20260203 21:07:02.260326 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:07:02.260582 410 external_mini_cluster.cc:1468] Setting key 1360990f447e9aeed35a03f584f7b7db
Feb 03 21:07:02 dist-test-slave-j889 krb5kdc[5057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.102.131@KRBTEST.COM; defaulting to no policy
I20260203 21:07:02.269101 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.258583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:45459 (local address 127.0.102.190:41819)
0203 21:07:02.258723 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:02.258728 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:02.259423 (+ 695us) server_negotiation.cc:408] Connection header received
0203 21:07:02.260287 (+ 864us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:02.260291 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:02.260349 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:02.260433 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:02.262112 (+ 1679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.262814 (+ 702us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.263436 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.263636 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.265753 (+ 2117us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:02.265773 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:02.265777 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:02.265807 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:02.267256 (+ 1449us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:02.267696 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:02.267700 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:02.267702 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:02.267756 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:02.268005 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:02.268009 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:02.268011 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:02.268204 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:02.268315 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:02.268813 (+ 498us) server_negotiation.cc:300] Negotiation successful
0203 21:07:02.268944 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260203 21:07:02.269706 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.258882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:41819 (local address 127.0.102.130:45459)
0203 21:07:02.259288 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:02.259324 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:02.260095 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:02.260577 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:02.260585 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:02.260991 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:02.261977 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:02.261992 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.262944 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.262947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:02.263334 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:02.263341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.263502 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.263996 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:02.264019 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:02.265607 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:02.267364 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:02.267368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:02.267377 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:02.267605 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:02.267834 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:02.267836 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:02.267838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:02.267925 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:02.268300 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:02.268305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:02.268544 (+ 239us) client_negotiation.cc:769] Sending connection context
0203 21:07:02.268746 (+ 202us) client_negotiation.cc:240] Negotiation successful
0203 21:07:02.268992 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":90,"threads_started":1}
I20260203 21:07:02.270790 5431 heartbeater.cc:344] Connected to a master server at 127.0.102.190:41819
I20260203 21:07:02.271024 5431 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:02.271476 5431 heartbeater.cc:507] Master 127.0.102.190:41819 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.131@KRBTEST.COM" created.
I20260203 21:07:02.272363 5104 ts_manager.cc:194] Registered new tserver with Master: 18ed75d8fa7d4431a3cb66847fba518f (127.0.102.130:41061)
I20260203 21:07:02.272943 5104 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.130@KRBTEST.COM'} at 127.0.102.130:45459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:07:02.301541 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:41819
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:02.402879 5438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:02.403066 5438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:02.403121 5438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:02.406234 5438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:02.406291 5438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:02.406368 5438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:07:02.410270 5438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:41819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:02.411209 5438 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:02.411986 5438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:02.417747 5446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:02.417728 5443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:02.417725 5444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:02.417905 5438 server_base.cc:1060] running on GCE node
I20260203 21:07:02.418291 5438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:02.418764 5438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:02.419898 5438 hybrid_clock.cc:648] HybridClock initialized: now 1770152822419885 us; error 26 us; skew 500 ppm
Feb 03 21:07:02 dist-test-slave-j889 krb5kdc[5057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:02.422282 5438 init.cc:377] Logged in from keytab as kudu/127.0.102.131@KRBTEST.COM (short username kudu)
I20260203 21:07:02.422925 5438 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:02.422979 5438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:02.423131 5438 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:02.424726 5438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "a7e72e1e9fd148f5aae32f2e92aa91a8"
format_stamp: "Formatted at 2026-02-03 21:07:02 on dist-test-slave-j889"
server_key: "75fd9da7edd6a23bbe0a9fb47605bd4f"
server_key_iv: "266f6b3acfcc1fbfde4afcef5210e74d"
server_key_version: "encryptionkey@0"
I20260203 21:07:02.425117 5438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "a7e72e1e9fd148f5aae32f2e92aa91a8"
format_stamp: "Formatted at 2026-02-03 21:07:02 on dist-test-slave-j889"
server_key: "75fd9da7edd6a23bbe0a9fb47605bd4f"
server_key_iv: "266f6b3acfcc1fbfde4afcef5210e74d"
server_key_version: "encryptionkey@0"
I20260203 21:07:02.428189 5438 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260203 21:07:02.430343 5452 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:02.431422 5438 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260203 21:07:02.431526 5438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "a7e72e1e9fd148f5aae32f2e92aa91a8"
format_stamp: "Formatted at 2026-02-03 21:07:02 on dist-test-slave-j889"
server_key: "75fd9da7edd6a23bbe0a9fb47605bd4f"
server_key_iv: "266f6b3acfcc1fbfde4afcef5210e74d"
server_key_version: "encryptionkey@0"
I20260203 21:07:02.431627 5438 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:02.454926 5438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:02.457458 5438 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:02.457615 5438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:02.458110 5438 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:02.458917 5438 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:02.458962 5438 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:02.459004 5438 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:02.459031 5438 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:02.467372 5438 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:34769
I20260203 21:07:02.467397 5565 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:34769 every 8 connection(s)
I20260203 21:07:02.467908 5438 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
Feb 03 21:07:02 dist-test-slave-j889 krb5kdc[5057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:07:02.475935 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 5438
I20260203 21:07:02.476064 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:07:02.476342 410 external_mini_cluster.cc:1468] Setting key 5fd7b78dc7fc88119420b59e5c2f9765
I20260203 21:07:02.478874 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.469547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:35801 (local address 127.0.102.190:41819)
0203 21:07:02.469693 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:02.469698 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:02.470364 (+ 666us) server_negotiation.cc:408] Connection header received
0203 21:07:02.471120 (+ 756us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:02.471123 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:02.471175 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:02.471243 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:02.472834 (+ 1591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.473327 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.473875 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.473992 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.475871 (+ 1879us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:02.475892 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:02.475895 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:02.475923 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:02.477278 (+ 1355us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:02.477696 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:02.477700 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:02.477701 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:02.477741 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:02.477984 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:02.477988 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:02.477990 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:02.478119 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:02.478197 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:02.478677 (+ 480us) server_negotiation.cc:300] Negotiation successful
0203 21:07:02.478767 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260203 21:07:02.479477 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.469819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:41819 (local address 127.0.102.131:35801)
0203 21:07:02.470222 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:02.470259 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:02.470953 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:02.471399 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:02.471407 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:02.471777 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:02.472725 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:02.472739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.473411 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.473414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:02.473795 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:02.473801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.473940 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.474414 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:02.474436 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:02.475738 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:02.477382 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:02.477387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:02.477397 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:02.477609 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:02.477814 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:02.477817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:02.477819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:02.477906 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:02.478208 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:02.478215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:02.478438 (+ 223us) client_negotiation.cc:769] Sending connection context
0203 21:07:02.478640 (+ 202us) client_negotiation.cc:240] Negotiation successful
0203 21:07:02.478830 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":92,"threads_started":1}
I20260203 21:07:02.480386 5566 heartbeater.cc:344] Connected to a master server at 127.0.102.190:41819
I20260203 21:07:02.480581 5566 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:02.481009 5566 heartbeater.cc:507] Master 127.0.102.190:41819 requested a full tablet report, sending...
I20260203 21:07:02.481850 5104 ts_manager.cc:194] Registered new tserver with Master: a7e72e1e9fd148f5aae32f2e92aa91a8 (127.0.102.131:34769)
I20260203 21:07:02.482348 5104 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.131@KRBTEST.COM'} at 127.0.102.131:35801
I20260203 21:07:02.489367 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260203 21:07:02.496459 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.490941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58030 (local address 127.0.102.190:41819)
0203 21:07:02.491052 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:02.491056 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:02.491200 (+ 144us) server_negotiation.cc:408] Connection header received
0203 21:07:02.491319 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:02.491322 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:02.491357 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:02.491413 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:02.492147 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.492608 (+ 461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.493198 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.493331 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.494200 (+ 869us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:02.494212 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:02.494214 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:02.494229 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:02.495241 (+ 1012us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:02.495550 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:02.495552 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:02.495553 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:02.495587 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:02.495752 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:02.495755 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:02.495757 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:02.495907 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:02.496048 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:02.496227 (+ 179us) server_negotiation.cc:300] Negotiation successful
0203 21:07:02.496327 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260203 21:07:02.498493 5104 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260203 21:07:02.500698 5104 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:07:02.511914 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.507837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:37815 (local address 127.0.0.1:48646)
0203 21:07:02.508169 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:02.508210 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:02.508360 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:02.509056 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:02.509062 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:02.509085 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:02.509390 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:02.509399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.510616 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.510619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:02.511442 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:02.511451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.511551 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.511605 (+ 54us) client_negotiation.cc:769] Sending connection context
0203 21:07:02.511689 (+ 84us) client_negotiation.cc:240] Negotiation successful
0203 21:07:02.511775 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
I20260203 21:07:02.512411 5580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.508306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:41061 (local address 127.0.0.1:39142)
0203 21:07:02.509390 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:02.509403 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:02.509485 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:02.509907 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:02.509909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:02.509926 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:02.510144 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:02.510153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.511155 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.511158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:02.512067 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:02.512076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.512180 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.512195 (+ 15us) client_negotiation.cc:769] Sending connection context
0203 21:07:02.512240 (+ 45us) client_negotiation.cc:240] Negotiation successful
0203 21:07:02.512303 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":110,"threads_started":1}
I20260203 21:07:02.512918 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.507948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48646 (local address 127.0.102.129:37815)
0203 21:07:02.508317 (+ 369us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:02.508323 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:02.508349 (+ 26us) server_negotiation.cc:408] Connection header received
0203 21:07:02.508508 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:02.508526 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:02.508735 (+ 209us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:02.508869 (+ 134us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:02.509550 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.510504 (+ 954us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.511899 (+ 1395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.512408 (+ 509us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.512539 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:02.512735 (+ 196us) server_negotiation.cc:300] Negotiation successful
0203 21:07:02.512807 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
I20260203 21:07:02.513105 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.508841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.0.102.130:41061)
0203 21:07:02.509347 (+ 506us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:02.509353 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:02.509473 (+ 120us) server_negotiation.cc:408] Connection header received
0203 21:07:02.509627 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:02.509632 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:02.509770 (+ 138us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:02.509888 (+ 118us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:02.510259 (+ 371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.511053 (+ 794us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.512207 (+ 1154us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.512748 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.512847 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:02.512903 (+ 56us) server_negotiation.cc:300] Negotiation successful
0203 21:07:02.512980 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":100,"threads_started":1}
I20260203 21:07:02.513100 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.508500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:34769 (local address 127.0.0.1:55320)
0203 21:07:02.509164 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:02.509178 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:02.509258 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:02.509745 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:02.509748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:02.509765 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:02.509992 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:02.509999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.511145 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.511149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:02.512792 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:02.512801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.512904 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.512917 (+ 13us) client_negotiation.cc:769] Sending connection context
0203 21:07:02.512962 (+ 45us) client_negotiation.cc:240] Negotiation successful
0203 21:07:02.513006 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":80,"threads_started":1}
I20260203 21:07:02.513763 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.508792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.0.102.131:34769)
0203 21:07:02.509150 (+ 358us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:02.509156 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:02.509370 (+ 214us) server_negotiation.cc:408] Connection header received
0203 21:07:02.509454 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:02.509460 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:02.509622 (+ 162us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:02.509754 (+ 132us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:02.510114 (+ 360us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.511032 (+ 918us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.512932 (+ 1900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.513426 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.513515 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:02.513570 (+ 55us) server_negotiation.cc:300] Negotiation successful
0203 21:07:02.513642 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":119,"threads_started":1}
I20260203 21:07:02.515694 5365 tablet_service.cc:1505] Processing CreateTablet for tablet c48d6f0baab948e2bef046a88614c74b (DEFAULT_TABLE table=test-table [id=9a8f30abe23a43699349c0534b40eabb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:02.515714 5230 tablet_service.cc:1505] Processing CreateTablet for tablet c48d6f0baab948e2bef046a88614c74b (DEFAULT_TABLE table=test-table [id=9a8f30abe23a43699349c0534b40eabb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:02.515938 5500 tablet_service.cc:1505] Processing CreateTablet for tablet c48d6f0baab948e2bef046a88614c74b (DEFAULT_TABLE table=test-table [id=9a8f30abe23a43699349c0534b40eabb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:02.516490 5365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c48d6f0baab948e2bef046a88614c74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:02.516568 5230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c48d6f0baab948e2bef046a88614c74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:02.516582 5500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c48d6f0baab948e2bef046a88614c74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:02.521492 5583 tablet_bootstrap.cc:492] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f: Bootstrap starting.
I20260203 21:07:02.522778 5584 tablet_bootstrap.cc:492] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8: Bootstrap starting.
I20260203 21:07:02.522815 5585 tablet_bootstrap.cc:492] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8: Bootstrap starting.
I20260203 21:07:02.523804 5583 tablet_bootstrap.cc:654] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:02.524464 5584 tablet_bootstrap.cc:654] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:02.524610 5585 tablet_bootstrap.cc:654] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:02.524605 5583 log.cc:826] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:02.525136 5584 log.cc:826] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:02.525302 5585 log.cc:826] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:02.526393 5583 tablet_bootstrap.cc:492] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f: No bootstrap required, opened a new log
I20260203 21:07:02.526572 5583 ts_tablet_manager.cc:1403] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260203 21:07:02.526695 5584 tablet_bootstrap.cc:492] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8: No bootstrap required, opened a new log
I20260203 21:07:02.526733 5585 tablet_bootstrap.cc:492] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8: No bootstrap required, opened a new log
I20260203 21:07:02.526855 5584 ts_tablet_manager.cc:1403] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260203 21:07:02.526899 5585 ts_tablet_manager.cc:1403] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260203 21:07:02.529707 5585 raft_consensus.cc:359] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37815 } } peers { permanent_uuid: "a7e72e1e9fd148f5aae32f2e92aa91a8" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34769 } } peers { permanent_uuid: "18ed75d8fa7d4431a3cb66847fba518f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41061 } }
I20260203 21:07:02.529878 5585 raft_consensus.cc:385] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:02.529951 5585 raft_consensus.cc:740] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f38e9e8daf14ef089ce0040f3ef5eb8, State: Initialized, Role: FOLLOWER
I20260203 21:07:02.530100 5584 raft_consensus.cc:359] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37815 } } peers { permanent_uuid: "a7e72e1e9fd148f5aae32f2e92aa91a8" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34769 } } peers { permanent_uuid: "18ed75d8fa7d4431a3cb66847fba518f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41061 } }
I20260203 21:07:02.530099 5583 raft_consensus.cc:359] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37815 } } peers { permanent_uuid: "a7e72e1e9fd148f5aae32f2e92aa91a8" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34769 } } peers { permanent_uuid: "18ed75d8fa7d4431a3cb66847fba518f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41061 } }
I20260203 21:07:02.530284 5584 raft_consensus.cc:385] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:02.530293 5583 raft_consensus.cc:385] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:02.530339 5583 raft_consensus.cc:740] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ed75d8fa7d4431a3cb66847fba518f, State: Initialized, Role: FOLLOWER
I20260203 21:07:02.530385 5584 raft_consensus.cc:740] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7e72e1e9fd148f5aae32f2e92aa91a8, State: Initialized, Role: FOLLOWER
I20260203 21:07:02.530380 5585 consensus_queue.cc:260] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37815 } } peers { permanent_uuid: "a7e72e1e9fd148f5aae32f2e92aa91a8" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34769 } } peers { permanent_uuid: "18ed75d8fa7d4431a3cb66847fba518f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41061 } }
I20260203 21:07:02.530794 5583 consensus_queue.cc:260] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37815 } } peers { permanent_uuid: "a7e72e1e9fd148f5aae32f2e92aa91a8" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34769 } } peers { permanent_uuid: "18ed75d8fa7d4431a3cb66847fba518f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41061 } }
I20260203 21:07:02.530794 5584 consensus_queue.cc:260] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37815 } } peers { permanent_uuid: "a7e72e1e9fd148f5aae32f2e92aa91a8" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34769 } } peers { permanent_uuid: "18ed75d8fa7d4431a3cb66847fba518f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41061 } }
I20260203 21:07:02.530997 5296 heartbeater.cc:499] Master 127.0.102.190:41819 was elected leader, sending a full tablet report...
I20260203 21:07:02.531208 5585 ts_tablet_manager.cc:1434] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260203 21:07:02.531565 5583 ts_tablet_manager.cc:1434] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260203 21:07:02.531569 5584 ts_tablet_manager.cc:1434] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260203 21:07:02.531903 5431 heartbeater.cc:499] Master 127.0.102.190:41819 was elected leader, sending a full tablet report...
I20260203 21:07:02.532087 5566 heartbeater.cc:499] Master 127.0.102.190:41819 was elected leader, sending a full tablet report...
W20260203 21:07:02.536528 5297 tablet.cc:2378] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260203 21:07:02.646677 5589 raft_consensus.cc:493] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:07:02.646849 5589 raft_consensus.cc:515] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37815 } } peers { permanent_uuid: "a7e72e1e9fd148f5aae32f2e92aa91a8" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34769 } } peers { permanent_uuid: "18ed75d8fa7d4431a3cb66847fba518f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41061 } }
I20260203 21:07:02.647884 5589 leader_election.cc:290] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7e72e1e9fd148f5aae32f2e92aa91a8 (127.0.102.131:34769), 18ed75d8fa7d4431a3cb66847fba518f (127.0.102.130:41061)
I20260203 21:07:02.650820 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.648100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:34769 (local address 127.0.102.129:50525)
0203 21:07:02.648418 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:02.648433 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:02.648555 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:02.648807 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:02.648810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:02.648825 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:02.649067 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:02.649073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.649779 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.649782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:02.650412 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:02.650420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.650520 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.650539 (+ 19us) client_negotiation.cc:769] Sending connection context
0203 21:07:02.650587 (+ 48us) client_negotiation.cc:240] Negotiation successful
0203 21:07:02.650644 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"spinlock_wait_cycles":6144,"thread_start_us":86,"threads_started":1}
I20260203 21:07:02.650835 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.648137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:41061 (local address 127.0.102.129:57897)
0203 21:07:02.648441 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:02.648452 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:02.648564 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:02.648812 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:02.648815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:02.648836 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:02.649081 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:02.649086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.649773 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.649776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:02.650407 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:02.650415 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.650517 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.650539 (+ 22us) client_negotiation.cc:769] Sending connection context
0203 21:07:02.650588 (+ 49us) client_negotiation.cc:240] Negotiation successful
0203 21:07:02.650644 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"spinlock_wait_cycles":10368,"thread_start_us":118,"threads_started":1}
I20260203 21:07:02.651254 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.648162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:50525 (local address 127.0.102.131:34769)
0203 21:07:02.648306 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:02.648311 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:02.648509 (+ 198us) server_negotiation.cc:408] Connection header received
0203 21:07:02.648676 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:02.648679 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:02.648722 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:02.648783 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:02.649179 (+ 396us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.649675 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.650521 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.650999 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.651037 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:02.651086 (+ 49us) server_negotiation.cc:300] Negotiation successful
0203 21:07:02.651143 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260203 21:07:02.651276 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.648241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:57897 (local address 127.0.102.130:41061)
0203 21:07:02.648401 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:02.648405 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:02.648543 (+ 138us) server_negotiation.cc:408] Connection header received
0203 21:07:02.648685 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:02.648687 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:02.648729 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:02.648789 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:02.649182 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.649670 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.650538 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.651020 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.651068 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:02.651118 (+ 50us) server_negotiation.cc:300] Negotiation successful
0203 21:07:02.651173 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260203 21:07:02.651765 5385 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c48d6f0baab948e2bef046a88614c74b" candidate_uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18ed75d8fa7d4431a3cb66847fba518f" is_pre_election: true
I20260203 21:07:02.651767 5520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c48d6f0baab948e2bef046a88614c74b" candidate_uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7e72e1e9fd148f5aae32f2e92aa91a8" is_pre_election: true
I20260203 21:07:02.651985 5385 raft_consensus.cc:2468] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f38e9e8daf14ef089ce0040f3ef5eb8 in term 0.
I20260203 21:07:02.651988 5520 raft_consensus.cc:2468] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f38e9e8daf14ef089ce0040f3ef5eb8 in term 0.
I20260203 21:07:02.652362 5184 leader_election.cc:304] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f38e9e8daf14ef089ce0040f3ef5eb8, a7e72e1e9fd148f5aae32f2e92aa91a8; no voters:
I20260203 21:07:02.652583 5589 raft_consensus.cc:2804] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:07:02.652674 5589 raft_consensus.cc:493] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:07:02.652724 5589 raft_consensus.cc:3060] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:02.653586 5589 raft_consensus.cc:515] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37815 } } peers { permanent_uuid: "a7e72e1e9fd148f5aae32f2e92aa91a8" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34769 } } peers { permanent_uuid: "18ed75d8fa7d4431a3cb66847fba518f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41061 } }
I20260203 21:07:02.653892 5589 leader_election.cc:290] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [CANDIDATE]: Term 1 election: Requested vote from peers a7e72e1e9fd148f5aae32f2e92aa91a8 (127.0.102.131:34769), 18ed75d8fa7d4431a3cb66847fba518f (127.0.102.130:41061)
I20260203 21:07:02.654179 5520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c48d6f0baab948e2bef046a88614c74b" candidate_uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7e72e1e9fd148f5aae32f2e92aa91a8"
I20260203 21:07:02.654238 5385 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c48d6f0baab948e2bef046a88614c74b" candidate_uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18ed75d8fa7d4431a3cb66847fba518f"
I20260203 21:07:02.654295 5520 raft_consensus.cc:3060] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:02.654345 5385 raft_consensus.cc:3060] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:02.655306 5520 raft_consensus.cc:2468] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f38e9e8daf14ef089ce0040f3ef5eb8 in term 1.
I20260203 21:07:02.655336 5385 raft_consensus.cc:2468] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f38e9e8daf14ef089ce0040f3ef5eb8 in term 1.
I20260203 21:07:02.655529 5184 leader_election.cc:304] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f38e9e8daf14ef089ce0040f3ef5eb8, a7e72e1e9fd148f5aae32f2e92aa91a8; no voters:
I20260203 21:07:02.655706 5589 raft_consensus.cc:2804] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:02.655898 5589 raft_consensus.cc:697] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [term 1 LEADER]: Becoming Leader. State: Replica: 8f38e9e8daf14ef089ce0040f3ef5eb8, State: Running, Role: LEADER
I20260203 21:07:02.656194 5589 consensus_queue.cc:237] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37815 } } peers { permanent_uuid: "a7e72e1e9fd148f5aae32f2e92aa91a8" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34769 } } peers { permanent_uuid: "18ed75d8fa7d4431a3cb66847fba518f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41061 } }
I20260203 21:07:02.658927 5103 catalog_manager.cc:5654] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f38e9e8daf14ef089ce0040f3ef5eb8 (127.0.102.129). New cstate: current_term: 1 leader_uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f38e9e8daf14ef089ce0040f3ef5eb8" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7e72e1e9fd148f5aae32f2e92aa91a8" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18ed75d8fa7d4431a3cb66847fba518f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41061 } health_report { overall_health: UNKNOWN } } }
I20260203 21:07:02.698132 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.695462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48656 (local address 127.0.102.129:37815)
0203 21:07:02.695595 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:02.695600 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:02.695633 (+ 33us) server_negotiation.cc:408] Connection header received
0203 21:07:02.695706 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:02.695709 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:02.695761 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:02.695831 (+ 70us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:07:02.696118 (+ 287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.696611 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:02.697248 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:02.697383 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:02.697462 (+ 79us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:07:02.697870 (+ 408us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:07:02.697956 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:02.697985 (+ 29us) server_negotiation.cc:300] Negotiation successful
0203 21:07:02.698029 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260203 21:07:02.703039 5385 raft_consensus.cc:1275] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f [term 1 FOLLOWER]: Refusing update from remote peer 8f38e9e8daf14ef089ce0040f3ef5eb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:02.703039 5520 raft_consensus.cc:1275] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8 [term 1 FOLLOWER]: Refusing update from remote peer 8f38e9e8daf14ef089ce0040f3ef5eb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:02.703645 5589 consensus_queue.cc:1048] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18ed75d8fa7d4431a3cb66847fba518f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:02.703801 5594 consensus_queue.cc:1048] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7e72e1e9fd148f5aae32f2e92aa91a8" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:02.710827 5598 mvcc.cc:204] Tried to move back new op lower bound from 7250545961785450496 to 7250545961602310144. Current Snapshot: MvccSnapshot[applied={T|T < 7250545961785450496}]
I20260203 21:07:02.720327 5103 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:07:02.720496 5103 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:07:02.722728 5103 catalog_manager.cc:5941] T 00000000000000000000000000000000 P d35980e2cb91482ba888b8895d52ab7f: Sending DeleteTablet for 3 replicas of tablet c48d6f0baab948e2bef046a88614c74b
I20260203 21:07:02.723393 5365 tablet_service.cc:1552] Processing DeleteTablet for tablet c48d6f0baab948e2bef046a88614c74b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:02 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:39142
I20260203 21:07:02.723441 5230 tablet_service.cc:1552] Processing DeleteTablet for tablet c48d6f0baab948e2bef046a88614c74b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:02 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:48646
I20260203 21:07:02.723562 5500 tablet_service.cc:1552] Processing DeleteTablet for tablet c48d6f0baab948e2bef046a88614c74b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:02 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:55320
I20260203 21:07:02.723845 5608 tablet_replica.cc:333] T c48d6f0baab948e2bef046a88614c74b P 8f38e9e8daf14ef089ce0040f3ef5eb8: stopping tablet replica
I20260203 21:07:02.723953 5609 tablet_replica.cc:333] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8: stopping tablet replica
I20260203 21:07:02.724059 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 5168
I20260203 21:07:02.724156 5609 raft_consensus.cc:2243] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:02.724411 5609 raft_consensus.cc:2272] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:02.725571 5609 ts_tablet_manager.cc:1916] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:02.725683 5607 tablet_replica.cc:333] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f: stopping tablet replica
I20260203 21:07:02.725912 5607 raft_consensus.cc:2243] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:02.726179 5607 raft_consensus.cc:2272] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:02.727413 5607 ts_tablet_manager.cc:1916] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:02.727573 5609 ts_tablet_manager.cc:1929] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:07:02.727648 5609 log.cc:1199] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-2/wal/wals/c48d6f0baab948e2bef046a88614c74b
I20260203 21:07:02.727913 5609 ts_tablet_manager.cc:1950] T c48d6f0baab948e2bef046a88614c74b P a7e72e1e9fd148f5aae32f2e92aa91a8: Deleting consensus metadata
I20260203 21:07:02.728842 5089 catalog_manager.cc:4985] TS a7e72e1e9fd148f5aae32f2e92aa91a8 (127.0.102.131:34769): tablet c48d6f0baab948e2bef046a88614c74b (table test-table [id=9a8f30abe23a43699349c0534b40eabb]) successfully deleted
I20260203 21:07:02.729714 5607 ts_tablet_manager.cc:1929] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:07:02.729804 5607 log.cc:1199] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1770152788143875-410-0/minicluster-data/ts-1/wal/wals/c48d6f0baab948e2bef046a88614c74b
I20260203 21:07:02.730091 5607 ts_tablet_manager.cc:1950] T c48d6f0baab948e2bef046a88614c74b P 18ed75d8fa7d4431a3cb66847fba518f: Deleting consensus metadata
I20260203 21:07:02.731097 5090 catalog_manager.cc:4985] TS 18ed75d8fa7d4431a3cb66847fba518f (127.0.102.130:41061): tablet c48d6f0baab948e2bef046a88614c74b (table test-table [id=9a8f30abe23a43699349c0534b40eabb]) successfully deleted
W20260203 21:07:02.731418 5089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.102.129:37815 (error 108)
I20260203 21:07:02.732021 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 5303
I20260203 21:07:02.732244 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.732004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:37815 (local address 127.0.0.1:48672)
0203 21:07:02.732117 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:02.732171 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.129:37815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260203 21:07:02.732512 5089 catalog_manager.cc:4712] TS 8f38e9e8daf14ef089ce0040f3ef5eb8 (127.0.102.129:37815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c48d6f0baab948e2bef046a88614c74b: Network error: Client connection negotiation failed: client connection to 127.0.102.129:37815: connect: Connection refused (error 111)
I20260203 21:07:02.737175 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 5438
I20260203 21:07:02.741453 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 5073
2026-02-03T21:07:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3153 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-02-03T21:07:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:07:02Z Disabled control of system clock
I20260203 21:07:02.767871 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:34041
--webserver_interface=127.0.102.190
--webserver_port=0
--builtin_ntp_servers=127.0.102.148:42265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:34041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260203 21:07:02.871263 5614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:02.871500 5614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:02.871546 5614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:02.874719 5614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:07:02.874775 5614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:02.874791 5614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:02.874814 5614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:07:02.874831 5614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:07:02.878954 5614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:42265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:34041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:34041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:02.879971 5614 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:02.881054 5614 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:02.886250 5622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:02.886234 5620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:02.886293 5614 server_base.cc:1060] running on GCE node
W20260203 21:07:02.886250 5619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:02.886998 5614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:02.887830 5614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:02.889008 5614 hybrid_clock.cc:648] HybridClock initialized: now 1770152822889002 us; error 26 us; skew 500 ppm
I20260203 21:07:02.890573 5614 webserver.cc:492] Webserver started at http://127.0.102.190:42039/ using document root <none> and password file <none>
I20260203 21:07:02.891068 5614 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:02.891124 5614 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:02.891283 5614 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:02.892922 5614 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "97f490f09d0a47788bd8bfb1c6d1be34"
format_stamp: "Formatted at 2026-02-03 21:07:02 on dist-test-slave-j889"
server_key: "70609aa849aaf3bd60a408dae390b410"
server_key_iv: "4c35c81c5e4acf6f74de325de1b1184f"
server_key_version: "encryptionkey@0"
I20260203 21:07:02.893347 5614 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "97f490f09d0a47788bd8bfb1c6d1be34"
format_stamp: "Formatted at 2026-02-03 21:07:02 on dist-test-slave-j889"
server_key: "70609aa849aaf3bd60a408dae390b410"
server_key_iv: "4c35c81c5e4acf6f74de325de1b1184f"
server_key_version: "encryptionkey@0"
I20260203 21:07:02.896353 5614 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260203 21:07:02.898383 5628 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:02.899278 5614 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260203 21:07:02.899385 5614 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "97f490f09d0a47788bd8bfb1c6d1be34"
format_stamp: "Formatted at 2026-02-03 21:07:02 on dist-test-slave-j889"
server_key: "70609aa849aaf3bd60a408dae390b410"
server_key_iv: "4c35c81c5e4acf6f74de325de1b1184f"
server_key_version: "encryptionkey@0"
I20260203 21:07:02.899466 5614 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:02.903836 5614 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:02.904302 5614 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:02.904436 5614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:02.910916 5614 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:34041
I20260203 21:07:02.910913 5680 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:34041 every 8 connection(s)
I20260203 21:07:02.911964 5614 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:07:02.914647 5681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:02.919881 5681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34: Bootstrap starting.
I20260203 21:07:02.922183 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 5614
I20260203 21:07:02.922290 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:07:02.922493 410 external_mini_cluster.cc:1468] Setting key 5a4ab0826380d9974a8e22f0c9ba9e3a
I20260203 21:07:02.922583 5681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:02.923403 5681 log.cc:826] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:02.925197 5681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34: No bootstrap required, opened a new log
I20260203 21:07:02.927129 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:02.923457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38086 (local address 127.0.102.190:34041)
0203 21:07:02.924180 (+ 723us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:02.924192 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:02.924230 (+ 38us) server_negotiation.cc:408] Connection header received
0203 21:07:02.924794 (+ 564us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:02.924812 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:02.925056 (+ 244us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:02.925350 (+ 294us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:02.926059 (+ 709us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:02.926063 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:02.926071 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:02.926081 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:02.926123 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:02.926134 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:02.926241 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:02.926351 (+ 110us) server_negotiation.cc:300] Negotiation successful
0203 21:07:02.926592 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":139,"threads_started":1}
I20260203 21:07:02.927775 5681 raft_consensus.cc:359] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f490f09d0a47788bd8bfb1c6d1be34" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 34041 } }
I20260203 21:07:02.927953 5681 raft_consensus.cc:385] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:02.928015 5681 raft_consensus.cc:740] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f490f09d0a47788bd8bfb1c6d1be34, State: Initialized, Role: FOLLOWER
I20260203 21:07:02.928447 5681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f490f09d0a47788bd8bfb1c6d1be34" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 34041 } }
I20260203 21:07:02.928586 5681 raft_consensus.cc:399] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:07:02.928666 5681 raft_consensus.cc:493] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:07:02.928752 5681 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:02.929610 5681 raft_consensus.cc:515] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f490f09d0a47788bd8bfb1c6d1be34" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 34041 } }
I20260203 21:07:02.929891 5681 leader_election.cc:304] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97f490f09d0a47788bd8bfb1c6d1be34; no voters:
I20260203 21:07:02.930124 5681 leader_election.cc:290] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:07:02.930224 5686 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:02.930413 5686 raft_consensus.cc:697] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [term 1 LEADER]: Becoming Leader. State: Replica: 97f490f09d0a47788bd8bfb1c6d1be34, State: Running, Role: LEADER
I20260203 21:07:02.930684 5686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f490f09d0a47788bd8bfb1c6d1be34" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 34041 } }
I20260203 21:07:02.931136 5681 sys_catalog.cc:565] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:07:02.931686 5688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97f490f09d0a47788bd8bfb1c6d1be34. Latest consensus state: current_term: 1 leader_uuid: "97f490f09d0a47788bd8bfb1c6d1be34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f490f09d0a47788bd8bfb1c6d1be34" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 34041 } } }
I20260203 21:07:02.931793 5688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:02.932001 5687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97f490f09d0a47788bd8bfb1c6d1be34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97f490f09d0a47788bd8bfb1c6d1be34" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 34041 } } }
I20260203 21:07:02.932085 5687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34 [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:02.932186 5693 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:07:02.934945 5693 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:07:02.939955 5693 catalog_manager.cc:1357] Generated new cluster ID: 3754a84fe0a742169e178a6acd3b4dfe
I20260203 21:07:02.940011 5693 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:07:02.959887 5693 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:07:02.960172 5693 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:07:02.971642 5693 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34: Generated new TSK 0
I20260203 21:07:02.972415 5693 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260203 21:07:02.981972 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:34041
--builtin_ntp_servers=127.0.102.148:42265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260203 21:07:03.099061 5705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:03.099332 5705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:03.099417 5705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:03.103859 5705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:03.103921 5705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:03.104023 5705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:07:03.108019 5705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:42265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:34041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:03.109315 5705 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:03.110486 5705 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:03.117478 5711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:03.117471 5710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:03.117758 5705 server_base.cc:1060] running on GCE node
W20260203 21:07:03.117795 5713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:03.118232 5705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:03.118798 5705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:03.119951 5705 hybrid_clock.cc:648] HybridClock initialized: now 1770152823119937 us; error 27 us; skew 500 ppm
I20260203 21:07:03.121846 5705 webserver.cc:492] Webserver started at http://127.0.102.129:34399/ using document root <none> and password file <none>
I20260203 21:07:03.122478 5705 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:03.122644 5705 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:03.122888 5705 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:03.124769 5705 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "a55a05657b8d406280b32938b4b1d8ea"
format_stamp: "Formatted at 2026-02-03 21:07:03 on dist-test-slave-j889"
server_key: "31e0353103ae27ca87b9a55cfd58133b"
server_key_iv: "4f46aa2bcfec8c96792dd0701fd6899d"
server_key_version: "encryptionkey@0"
I20260203 21:07:03.125239 5705 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "a55a05657b8d406280b32938b4b1d8ea"
format_stamp: "Formatted at 2026-02-03 21:07:03 on dist-test-slave-j889"
server_key: "31e0353103ae27ca87b9a55cfd58133b"
server_key_iv: "4f46aa2bcfec8c96792dd0701fd6899d"
server_key_version: "encryptionkey@0"
I20260203 21:07:03.128587 5705 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260203 21:07:03.130904 5719 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:03.131922 5705 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260203 21:07:03.132036 5705 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "a55a05657b8d406280b32938b4b1d8ea"
format_stamp: "Formatted at 2026-02-03 21:07:03 on dist-test-slave-j889"
server_key: "31e0353103ae27ca87b9a55cfd58133b"
server_key_iv: "4f46aa2bcfec8c96792dd0701fd6899d"
server_key_version: "encryptionkey@0"
I20260203 21:07:03.132164 5705 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:03.137043 5705 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:03.137617 5705 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:03.137768 5705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:03.138250 5705 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:03.139262 5705 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:03.139328 5705 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:03.139372 5705 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:03.139405 5705 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:03.148988 5705 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:36243
I20260203 21:07:03.149029 5832 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:36243 every 8 connection(s)
I20260203 21:07:03.149993 5705 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:07:03.155272 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.151820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:59745 (local address 127.0.102.190:34041)
0203 21:07:03.151968 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:03.151972 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:03.152867 (+ 895us) server_negotiation.cc:408] Connection header received
0203 21:07:03.153666 (+ 799us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:03.153670 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:03.153715 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:03.153793 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:03.154818 (+ 1025us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:03.154822 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:03.154825 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:03.154838 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:03.154858 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:03.154866 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:03.154944 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:03.155076 (+ 132us) server_negotiation.cc:300] Negotiation successful
0203 21:07:03.155119 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260203 21:07:03.155989 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.152066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34041 (local address 127.0.102.129:59745)
0203 21:07:03.152740 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:03.152774 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:03.153471 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:03.153951 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:03.153959 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:03.154045 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:03.154583 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0203 21:07:03.154585 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:03.154620 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0203 21:07:03.154675 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:03.154936 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:03.154943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:03.154948 (+ 5us) client_negotiation.cc:769] Sending connection context
0203 21:07:03.155139 (+ 191us) client_negotiation.cc:240] Negotiation successful
0203 21:07:03.155398 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":98,"threads_started":1}
I20260203 21:07:03.157186 5833 heartbeater.cc:344] Connected to a master server at 127.0.102.190:34041
I20260203 21:07:03.157445 5833 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:03.157768 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 5705
I20260203 21:07:03.157832 5833 heartbeater.cc:507] Master 127.0.102.190:34041 requested a full tablet report, sending...
I20260203 21:07:03.157873 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:07:03.158104 410 external_mini_cluster.cc:1468] Setting key 1bca1f1b29840de0ad938f76d7723911
I20260203 21:07:03.159250 5645 ts_manager.cc:194] Registered new tserver with Master: a55a05657b8d406280b32938b4b1d8ea (127.0.102.129:36243)
I20260203 21:07:03.160156 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:34041
--builtin_ntp_servers=127.0.102.148:42265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260203 21:07:03.263056 5836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:03.263320 5836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:03.263401 5836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:03.266737 5836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:03.266805 5836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:03.266912 5836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:07:03.270813 5836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:42265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:34041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:03.271909 5836 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:03.273082 5836 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:03.280177 5841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:03.280162 5844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:03.280162 5842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:03.280494 5836 server_base.cc:1060] running on GCE node
I20260203 21:07:03.280849 5836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:03.281339 5836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:03.282503 5836 hybrid_clock.cc:648] HybridClock initialized: now 1770152823282492 us; error 29 us; skew 500 ppm
I20260203 21:07:03.284261 5836 webserver.cc:492] Webserver started at http://127.0.102.130:35551/ using document root <none> and password file <none>
I20260203 21:07:03.284878 5836 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:03.284934 5836 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:03.285105 5836 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:03.286738 5836 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "932b6455e731419fb5747ab9d59aca8d"
format_stamp: "Formatted at 2026-02-03 21:07:03 on dist-test-slave-j889"
server_key: "2dfaf90d91443747655977baa221dd14"
server_key_iv: "949da60af6d410fb09fce5a3de67b29a"
server_key_version: "encryptionkey@0"
I20260203 21:07:03.287173 5836 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "932b6455e731419fb5747ab9d59aca8d"
format_stamp: "Formatted at 2026-02-03 21:07:03 on dist-test-slave-j889"
server_key: "2dfaf90d91443747655977baa221dd14"
server_key_iv: "949da60af6d410fb09fce5a3de67b29a"
server_key_version: "encryptionkey@0"
I20260203 21:07:03.290416 5836 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260203 21:07:03.292707 5850 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:03.293731 5836 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:07:03.293836 5836 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "932b6455e731419fb5747ab9d59aca8d"
format_stamp: "Formatted at 2026-02-03 21:07:03 on dist-test-slave-j889"
server_key: "2dfaf90d91443747655977baa221dd14"
server_key_iv: "949da60af6d410fb09fce5a3de67b29a"
server_key_version: "encryptionkey@0"
I20260203 21:07:03.293920 5836 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:03.298484 5836 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:03.299019 5836 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:03.299163 5836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:03.299708 5836 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:03.300756 5836 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:03.300808 5836 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:03.300858 5836 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:03.300887 5836 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:03.309209 5836 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:37355
I20260203 21:07:03.309237 5963 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:37355 every 8 connection(s)
I20260203 21:07:03.310115 5836 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:07:03.315318 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 5836
I20260203 21:07:03.315375 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.311807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:49267 (local address 127.0.102.190:34041)
0203 21:07:03.311945 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:03.311949 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:03.312875 (+ 926us) server_negotiation.cc:408] Connection header received
0203 21:07:03.313651 (+ 776us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:03.313654 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:03.313702 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:03.313777 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:03.314887 (+ 1110us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:03.314891 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:03.314894 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:03.314908 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:03.314929 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:03.314937 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:03.315019 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:03.315152 (+ 133us) server_negotiation.cc:300] Negotiation successful
0203 21:07:03.315210 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260203 21:07:03.315440 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:07:03.315704 410 external_mini_cluster.cc:1468] Setting key 07d0d327bb6e1d6d4f735d90880bf73e
I20260203 21:07:03.316689 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.312089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34041 (local address 127.0.102.130:49267)
0203 21:07:03.312742 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:03.312776 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:03.313457 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:03.313943 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:03.313951 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:03.314035 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:03.314636 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0203 21:07:03.314638 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:03.314670 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0203 21:07:03.314724 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:03.315009 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:03.315016 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:03.315021 (+ 5us) client_negotiation.cc:769] Sending connection context
0203 21:07:03.315221 (+ 200us) client_negotiation.cc:240] Negotiation successful
0203 21:07:03.315433 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":99,"threads_started":1}
I20260203 21:07:03.317518 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:34041
--builtin_ntp_servers=127.0.102.148:42265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260203 21:07:03.317713 5964 heartbeater.cc:344] Connected to a master server at 127.0.102.190:34041
I20260203 21:07:03.317927 5964 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:03.318404 5964 heartbeater.cc:507] Master 127.0.102.190:34041 requested a full tablet report, sending...
I20260203 21:07:03.319425 5645 ts_manager.cc:194] Registered new tserver with Master: 932b6455e731419fb5747ab9d59aca8d (127.0.102.130:37355)
W20260203 21:07:03.419157 5967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:03.419373 5967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:03.419423 5967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:03.422634 5967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:03.422684 5967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:03.422757 5967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:07:03.426539 5967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:42265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:34041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:03.427574 5967 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:03.428696 5967 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:03.435163 5973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:03.435139 5972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:03.435272 5967 server_base.cc:1060] running on GCE node
W20260203 21:07:03.435115 5975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:03.435865 5967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:03.436434 5967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:03.437592 5967 hybrid_clock.cc:648] HybridClock initialized: now 1770152823437564 us; error 40 us; skew 500 ppm
I20260203 21:07:03.439275 5967 webserver.cc:492] Webserver started at http://127.0.102.131:34671/ using document root <none> and password file <none>
I20260203 21:07:03.439831 5967 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:03.439895 5967 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:03.440092 5967 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:03.441746 5967 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "2e9c1e9456914a6f88d8b29e272b529b"
format_stamp: "Formatted at 2026-02-03 21:07:03 on dist-test-slave-j889"
server_key: "2b3b7fe777934e43e5279ac23802b610"
server_key_iv: "f9d9366a084859b071b0bcf924b981bf"
server_key_version: "encryptionkey@0"
I20260203 21:07:03.442222 5967 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "2e9c1e9456914a6f88d8b29e272b529b"
format_stamp: "Formatted at 2026-02-03 21:07:03 on dist-test-slave-j889"
server_key: "2b3b7fe777934e43e5279ac23802b610"
server_key_iv: "f9d9366a084859b071b0bcf924b981bf"
server_key_version: "encryptionkey@0"
I20260203 21:07:03.445441 5967 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260203 21:07:03.447532 5981 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:03.448474 5967 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:03.448571 5967 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "2e9c1e9456914a6f88d8b29e272b529b"
format_stamp: "Formatted at 2026-02-03 21:07:03 on dist-test-slave-j889"
server_key: "2b3b7fe777934e43e5279ac23802b610"
server_key_iv: "f9d9366a084859b071b0bcf924b981bf"
server_key_version: "encryptionkey@0"
I20260203 21:07:03.448693 5967 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:03.453285 5967 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:03.453828 5967 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:03.453972 5967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:03.454421 5967 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:03.455276 5967 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:03.455322 5967 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:03.455355 5967 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:03.455382 5967 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:03.463799 5967 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:46377
I20260203 21:07:03.463832 6094 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:46377 every 8 connection(s)
I20260203 21:07:03.464742 5967 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
I20260203 21:07:03.469750 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.466487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:52197 (local address 127.0.102.190:34041)
0203 21:07:03.466621 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:03.466624 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:03.467495 (+ 871us) server_negotiation.cc:408] Connection header received
0203 21:07:03.468217 (+ 722us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:03.468220 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:03.468259 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:03.468331 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:03.469282 (+ 951us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:03.469285 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:03.469287 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:03.469300 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:03.469321 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:03.469329 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:03.469407 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:03.469598 (+ 191us) server_negotiation.cc:300] Negotiation successful
0203 21:07:03.469632 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260203 21:07:03.470466 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.466739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34041 (local address 127.0.102.131:52197)
0203 21:07:03.467339 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:03.467371 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:03.468050 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:03.468485 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:03.468492 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:03.468572 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:03.469062 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0203 21:07:03.469064 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:03.469096 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0203 21:07:03.469146 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:03.469457 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:03.469465 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:03.469469 (+ 4us) client_negotiation.cc:769] Sending connection context
0203 21:07:03.469675 (+ 206us) client_negotiation.cc:240] Negotiation successful
0203 21:07:03.469866 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":108,"threads_started":1}
I20260203 21:07:03.471807 6095 heartbeater.cc:344] Connected to a master server at 127.0.102.190:34041
I20260203 21:07:03.472034 6095 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:03.472429 6095 heartbeater.cc:507] Master 127.0.102.190:34041 requested a full tablet report, sending...
I20260203 21:07:03.472733 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 5967
I20260203 21:07:03.472852 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:07:03.473127 410 external_mini_cluster.cc:1468] Setting key 011155cd5db96469cf0db0e812289c3a
I20260203 21:07:03.473968 5644 ts_manager.cc:194] Registered new tserver with Master: 2e9c1e9456914a6f88d8b29e272b529b (127.0.102.131:46377)
I20260203 21:07:03.475024 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260203 21:07:03.478160 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.476728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.0.102.190:34041)
0203 21:07:03.476848 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:03.476852 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:03.476966 (+ 114us) server_negotiation.cc:408] Connection header received
0203 21:07:03.477088 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:03.477091 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:03.477123 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:03.477181 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:03.477825 (+ 644us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:03.477829 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:03.477832 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:03.477843 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:03.477864 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:03.477872 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:03.477939 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:03.478010 (+ 71us) server_negotiation.cc:300] Negotiation successful
0203 21:07:03.478044 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260203 21:07:03.480404 5645 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260203 21:07:03.482677 5645 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:07:03.493134 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.490321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:36243 (local address 127.0.0.1:53104)
0203 21:07:03.490643 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:03.490655 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:03.490734 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:03.491315 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:03.491320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:03.491383 (+ 63us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:03.491909 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0203 21:07:03.491911 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:03.491955 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0203 21:07:03.492020 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:03.492396 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:03.492404 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:03.492409 (+ 5us) client_negotiation.cc:769] Sending connection context
0203 21:07:03.492943 (+ 534us) client_negotiation.cc:240] Negotiation successful
0203 21:07:03.493000 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":90,"threads_started":1}
I20260203 21:07:03.493352 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.489858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:46377 (local address 127.0.0.1:42980)
0203 21:07:03.490159 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:03.490196 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:03.490289 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:03.492268 (+ 1979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:03.492271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:03.492286 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:03.492784 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0203 21:07:03.492786 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:03.492794 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0203 21:07:03.492814 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:03.493111 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:03.493114 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:03.493117 (+ 3us) client_negotiation.cc:769] Sending connection context
0203 21:07:03.493189 (+ 72us) client_negotiation.cc:240] Negotiation successful
0203 21:07:03.493221 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260203 21:07:03.493538 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.490541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.0.102.129:36243)
0203 21:07:03.490960 (+ 419us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:03.490965 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:03.490987 (+ 22us) server_negotiation.cc:408] Connection header received
0203 21:07:03.491037 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:03.491041 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:03.491159 (+ 118us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:03.491289 (+ 130us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:03.492153 (+ 864us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:03.492158 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:03.492168 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:03.492181 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:03.492241 (+ 60us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:03.492255 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:03.492352 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:03.493315 (+ 963us) server_negotiation.cc:300] Negotiation successful
0203 21:07:03.493387 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":130,"threads_started":1}
I20260203 21:07:03.493366 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.490449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:37355 (local address 127.0.0.1:50358)
0203 21:07:03.490904 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:03.490916 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:03.490977 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:03.492297 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:03.492300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:03.492310 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:03.492859 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0203 21:07:03.492861 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:03.492870 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0203 21:07:03.492888 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:03.493193 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:03.493196 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:03.493199 (+ 3us) client_negotiation.cc:769] Sending connection context
0203 21:07:03.493256 (+ 57us) client_negotiation.cc:240] Negotiation successful
0203 21:07:03.493280 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":67,"threads_started":1}
I20260203 21:07:03.493645 6111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.490791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.0.102.130:37355)
0203 21:07:03.491972 (+ 1181us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:03.491977 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:03.491996 (+ 19us) server_negotiation.cc:408] Connection header received
0203 21:07:03.492036 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:03.492040 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:03.492170 (+ 130us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:03.492306 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:03.492981 (+ 675us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:03.492987 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:03.492996 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:03.493008 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:03.493051 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:03.493060 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:03.493157 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:03.493432 (+ 275us) server_negotiation.cc:300] Negotiation successful
0203 21:07:03.493500 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1095,"thread_start_us":95,"threads_started":1}
I20260203 21:07:03.494081 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.489931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.0.102.131:46377)
0203 21:07:03.490281 (+ 350us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:03.490286 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:03.490309 (+ 23us) server_negotiation.cc:408] Connection header received
0203 21:07:03.490453 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:03.490459 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:03.490618 (+ 159us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:03.491262 (+ 644us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:03.492935 (+ 1673us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:03.492941 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:03.492951 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:03.492963 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:03.493017 (+ 54us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:03.493029 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:03.493132 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:03.493878 (+ 746us) server_negotiation.cc:300] Negotiation successful
0203 21:07:03.493941 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":99,"threads_started":1}
I20260203 21:07:03.495896 5767 tablet_service.cc:1505] Processing CreateTablet for tablet 27ae008ddbe649ec84824d081ce78c3e (DEFAULT_TABLE table=test-table [id=1b07bfe6c10446fcb45252cc3793dacf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:03.496132 6029 tablet_service.cc:1505] Processing CreateTablet for tablet 27ae008ddbe649ec84824d081ce78c3e (DEFAULT_TABLE table=test-table [id=1b07bfe6c10446fcb45252cc3793dacf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:03.496058 5898 tablet_service.cc:1505] Processing CreateTablet for tablet 27ae008ddbe649ec84824d081ce78c3e (DEFAULT_TABLE table=test-table [id=1b07bfe6c10446fcb45252cc3793dacf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:03.496799 5767 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ae008ddbe649ec84824d081ce78c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:03.496945 6029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ae008ddbe649ec84824d081ce78c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:03.496945 5898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ae008ddbe649ec84824d081ce78c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:03.501986 6112 tablet_bootstrap.cc:492] T 27ae008ddbe649ec84824d081ce78c3e P a55a05657b8d406280b32938b4b1d8ea: Bootstrap starting.
I20260203 21:07:03.503376 6113 tablet_bootstrap.cc:492] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d: Bootstrap starting.
I20260203 21:07:03.503532 6114 tablet_bootstrap.cc:492] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b: Bootstrap starting.
I20260203 21:07:03.504323 6112 tablet_bootstrap.cc:654] T 27ae008ddbe649ec84824d081ce78c3e P a55a05657b8d406280b32938b4b1d8ea: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:03.505049 6112 log.cc:826] T 27ae008ddbe649ec84824d081ce78c3e P a55a05657b8d406280b32938b4b1d8ea: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:03.505520 6113 tablet_bootstrap.cc:654] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:03.505519 6114 tablet_bootstrap.cc:654] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:03.506297 6113 log.cc:826] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:03.506297 6114 log.cc:826] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:03.506717 6112 tablet_bootstrap.cc:492] T 27ae008ddbe649ec84824d081ce78c3e P a55a05657b8d406280b32938b4b1d8ea: No bootstrap required, opened a new log
I20260203 21:07:03.506914 6112 ts_tablet_manager.cc:1403] T 27ae008ddbe649ec84824d081ce78c3e P a55a05657b8d406280b32938b4b1d8ea: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260203 21:07:03.508073 6113 tablet_bootstrap.cc:492] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d: No bootstrap required, opened a new log
I20260203 21:07:03.508312 6113 ts_tablet_manager.cc:1403] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260203 21:07:03.508409 6114 tablet_bootstrap.cc:492] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b: No bootstrap required, opened a new log
I20260203 21:07:03.508601 6114 ts_tablet_manager.cc:1403] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260203 21:07:03.509696 6112 raft_consensus.cc:359] T 27ae008ddbe649ec84824d081ce78c3e P a55a05657b8d406280b32938b4b1d8ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e9c1e9456914a6f88d8b29e272b529b" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46377 } } peers { permanent_uuid: "932b6455e731419fb5747ab9d59aca8d" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 37355 } } peers { permanent_uuid: "a55a05657b8d406280b32938b4b1d8ea" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36243 } }
I20260203 21:07:03.509886 6112 raft_consensus.cc:385] T 27ae008ddbe649ec84824d081ce78c3e P a55a05657b8d406280b32938b4b1d8ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:03.509955 6112 raft_consensus.cc:740] T 27ae008ddbe649ec84824d081ce78c3e P a55a05657b8d406280b32938b4b1d8ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55a05657b8d406280b32938b4b1d8ea, State: Initialized, Role: FOLLOWER
I20260203 21:07:03.510345 6112 consensus_queue.cc:260] T 27ae008ddbe649ec84824d081ce78c3e P a55a05657b8d406280b32938b4b1d8ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e9c1e9456914a6f88d8b29e272b529b" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46377 } } peers { permanent_uuid: "932b6455e731419fb5747ab9d59aca8d" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 37355 } } peers { permanent_uuid: "a55a05657b8d406280b32938b4b1d8ea" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36243 } }
I20260203 21:07:03.510941 5833 heartbeater.cc:499] Master 127.0.102.190:34041 was elected leader, sending a full tablet report...
I20260203 21:07:03.511178 6112 ts_tablet_manager.cc:1434] T 27ae008ddbe649ec84824d081ce78c3e P a55a05657b8d406280b32938b4b1d8ea: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260203 21:07:03.511426 6114 raft_consensus.cc:359] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e9c1e9456914a6f88d8b29e272b529b" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46377 } } peers { permanent_uuid: "932b6455e731419fb5747ab9d59aca8d" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 37355 } } peers { permanent_uuid: "a55a05657b8d406280b32938b4b1d8ea" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36243 } }
I20260203 21:07:03.511420 6113 raft_consensus.cc:359] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e9c1e9456914a6f88d8b29e272b529b" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46377 } } peers { permanent_uuid: "932b6455e731419fb5747ab9d59aca8d" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 37355 } } peers { permanent_uuid: "a55a05657b8d406280b32938b4b1d8ea" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36243 } }
I20260203 21:07:03.511620 6114 raft_consensus.cc:385] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:03.511620 6113 raft_consensus.cc:385] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:03.511684 6113 raft_consensus.cc:740] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932b6455e731419fb5747ab9d59aca8d, State: Initialized, Role: FOLLOWER
I20260203 21:07:03.511699 6114 raft_consensus.cc:740] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e9c1e9456914a6f88d8b29e272b529b, State: Initialized, Role: FOLLOWER
I20260203 21:07:03.512112 6114 consensus_queue.cc:260] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e9c1e9456914a6f88d8b29e272b529b" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46377 } } peers { permanent_uuid: "932b6455e731419fb5747ab9d59aca8d" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 37355 } } peers { permanent_uuid: "a55a05657b8d406280b32938b4b1d8ea" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36243 } }
I20260203 21:07:03.512112 6113 consensus_queue.cc:260] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e9c1e9456914a6f88d8b29e272b529b" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46377 } } peers { permanent_uuid: "932b6455e731419fb5747ab9d59aca8d" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 37355 } } peers { permanent_uuid: "a55a05657b8d406280b32938b4b1d8ea" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36243 } }
I20260203 21:07:03.512967 6113 ts_tablet_manager.cc:1434] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260203 21:07:03.512967 6114 ts_tablet_manager.cc:1434] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260203 21:07:03.512995 5964 heartbeater.cc:499] Master 127.0.102.190:34041 was elected leader, sending a full tablet report...
I20260203 21:07:03.513159 6095 heartbeater.cc:499] Master 127.0.102.190:34041 was elected leader, sending a full tablet report...
W20260203 21:07:03.560982 5965 tablet.cc:2378] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260203 21:07:03.567413 6120 raft_consensus.cc:493] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:07:03.567600 6120 raft_consensus.cc:515] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e9c1e9456914a6f88d8b29e272b529b" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46377 } } peers { permanent_uuid: "932b6455e731419fb5747ab9d59aca8d" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 37355 } } peers { permanent_uuid: "a55a05657b8d406280b32938b4b1d8ea" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36243 } }
I20260203 21:07:03.568943 6120 leader_election.cc:290] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 932b6455e731419fb5747ab9d59aca8d (127.0.102.130:37355), a55a05657b8d406280b32938b4b1d8ea (127.0.102.129:36243)
I20260203 21:07:03.570688 6111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.569245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:48535 (local address 127.0.102.130:37355)
0203 21:07:03.569372 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:03.569376 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:03.569390 (+ 14us) server_negotiation.cc:408] Connection header received
0203 21:07:03.569445 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:03.569448 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:03.569487 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:03.569566 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:03.570256 (+ 690us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:03.570260 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:03.570262 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:03.570273 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:03.570293 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:03.570300 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:03.570370 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:03.570535 (+ 165us) server_negotiation.cc:300] Negotiation successful
0203 21:07:03.570572 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260203 21:07:03.570721 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.569120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:37355 (local address 127.0.102.131:48535)
0203 21:07:03.569237 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:03.569254 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:03.569337 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:03.569591 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:03.569594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:03.569608 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:03.570088 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0203 21:07:03.570090 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:03.570104 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0203 21:07:03.570129 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:03.570430 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:03.570434 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:03.570438 (+ 4us) client_negotiation.cc:769] Sending connection context
0203 21:07:03.570541 (+ 103us) client_negotiation.cc:240] Negotiation successful
0203 21:07:03.570587 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260203 21:07:03.571286 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.569465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:34105 (local address 127.0.102.129:36243)
0203 21:07:03.569599 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:03.569605 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:03.569654 (+ 49us) server_negotiation.cc:408] Connection header received
0203 21:07:03.569770 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:03.569774 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:03.569812 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:03.569886 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:03.570868 (+ 982us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:03.570873 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:03.570876 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:03.570890 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:03.570918 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:03.570929 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:03.570995 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:03.571156 (+ 161us) server_negotiation.cc:300] Negotiation successful
0203 21:07:03.571193 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260203 21:07:03.571341 6121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.569194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:36243 (local address 127.0.102.131:34105)
0203 21:07:03.569579 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:03.569593 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:03.569682 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:03.569911 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:03.569913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:03.569926 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:03.570682 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0203 21:07:03.570684 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:03.570695 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0203 21:07:03.570718 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:03.571058 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:03.571062 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:03.571065 (+ 3us) client_negotiation.cc:769] Sending connection context
0203 21:07:03.571140 (+ 75us) client_negotiation.cc:240] Negotiation successful
0203 21:07:03.571175 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":114,"threads_started":1}
I20260203 21:07:03.571382 5918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27ae008ddbe649ec84824d081ce78c3e" candidate_uuid: "2e9c1e9456914a6f88d8b29e272b529b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "932b6455e731419fb5747ab9d59aca8d" is_pre_election: true
I20260203 21:07:03.571678 5918 raft_consensus.cc:2468] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e9c1e9456914a6f88d8b29e272b529b in term 0.
I20260203 21:07:03.571916 5787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27ae008ddbe649ec84824d081ce78c3e" candidate_uuid: "2e9c1e9456914a6f88d8b29e272b529b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55a05657b8d406280b32938b4b1d8ea" is_pre_election: true
I20260203 21:07:03.572163 5787 raft_consensus.cc:2468] T 27ae008ddbe649ec84824d081ce78c3e P a55a05657b8d406280b32938b4b1d8ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e9c1e9456914a6f88d8b29e272b529b in term 0.
I20260203 21:07:03.572168 5985 leader_election.cc:304] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e9c1e9456914a6f88d8b29e272b529b, 932b6455e731419fb5747ab9d59aca8d; no voters:
I20260203 21:07:03.572393 6120 raft_consensus.cc:2804] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:07:03.572580 6120 raft_consensus.cc:493] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:07:03.572697 6120 raft_consensus.cc:3060] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:03.573653 6120 raft_consensus.cc:515] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e9c1e9456914a6f88d8b29e272b529b" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46377 } } peers { permanent_uuid: "932b6455e731419fb5747ab9d59aca8d" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 37355 } } peers { permanent_uuid: "a55a05657b8d406280b32938b4b1d8ea" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36243 } }
I20260203 21:07:03.573982 6120 leader_election.cc:290] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [CANDIDATE]: Term 1 election: Requested vote from peers 932b6455e731419fb5747ab9d59aca8d (127.0.102.130:37355), a55a05657b8d406280b32938b4b1d8ea (127.0.102.129:36243)
I20260203 21:07:03.574280 5918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27ae008ddbe649ec84824d081ce78c3e" candidate_uuid: "2e9c1e9456914a6f88d8b29e272b529b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "932b6455e731419fb5747ab9d59aca8d"
I20260203 21:07:03.574283 5787 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27ae008ddbe649ec84824d081ce78c3e" candidate_uuid: "2e9c1e9456914a6f88d8b29e272b529b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55a05657b8d406280b32938b4b1d8ea"
I20260203 21:07:03.574381 5787 raft_consensus.cc:3060] T 27ae008ddbe649ec84824d081ce78c3e P a55a05657b8d406280b32938b4b1d8ea [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:03.574393 5918 raft_consensus.cc:3060] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:03.575660 5918 raft_consensus.cc:2468] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e9c1e9456914a6f88d8b29e272b529b in term 1.
I20260203 21:07:03.575646 5787 raft_consensus.cc:2468] T 27ae008ddbe649ec84824d081ce78c3e P a55a05657b8d406280b32938b4b1d8ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e9c1e9456914a6f88d8b29e272b529b in term 1.
I20260203 21:07:03.575958 5984 leader_election.cc:304] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e9c1e9456914a6f88d8b29e272b529b, a55a05657b8d406280b32938b4b1d8ea; no voters:
I20260203 21:07:03.576109 6120 raft_consensus.cc:2804] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:03.576350 6120 raft_consensus.cc:697] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [term 1 LEADER]: Becoming Leader. State: Replica: 2e9c1e9456914a6f88d8b29e272b529b, State: Running, Role: LEADER
I20260203 21:07:03.576666 6120 consensus_queue.cc:237] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e9c1e9456914a6f88d8b29e272b529b" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46377 } } peers { permanent_uuid: "932b6455e731419fb5747ab9d59aca8d" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 37355 } } peers { permanent_uuid: "a55a05657b8d406280b32938b4b1d8ea" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36243 } }
I20260203 21:07:03.579464 5645 catalog_manager.cc:5654] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e9c1e9456914a6f88d8b29e272b529b (127.0.102.131). New cstate: current_term: 1 leader_uuid: "2e9c1e9456914a6f88d8b29e272b529b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e9c1e9456914a6f88d8b29e272b529b" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 46377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "932b6455e731419fb5747ab9d59aca8d" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 37355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a55a05657b8d406280b32938b4b1d8ea" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36243 } health_report { overall_health: UNKNOWN } } }
I20260203 21:07:03.593298 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.591962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.0.102.131:46377)
0203 21:07:03.592078 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:03.592081 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:03.592099 (+ 18us) server_negotiation.cc:408] Connection header received
0203 21:07:03.592172 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:03.592175 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:03.592229 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:03.592299 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:03.592975 (+ 676us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:03.592978 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:03.592980 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:03.592991 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:03.593009 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:03.593016 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:03.593084 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:03.593138 (+ 54us) server_negotiation.cc:300] Negotiation successful
0203 21:07:03.593169 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260203 21:07:03.598412 5787 raft_consensus.cc:1275] T 27ae008ddbe649ec84824d081ce78c3e P a55a05657b8d406280b32938b4b1d8ea [term 1 FOLLOWER]: Refusing update from remote peer 2e9c1e9456914a6f88d8b29e272b529b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:03.598404 5918 raft_consensus.cc:1275] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d [term 1 FOLLOWER]: Refusing update from remote peer 2e9c1e9456914a6f88d8b29e272b529b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:03.598963 6120 consensus_queue.cc:1048] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55a05657b8d406280b32938b4b1d8ea" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:03.599120 6122 consensus_queue.cc:1048] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [LEADER]: Connected to new peer: Peer: permanent_uuid: "932b6455e731419fb5747ab9d59aca8d" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 37355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:03.614178 5643 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:07:03.614358 5643 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:07:03.616869 5643 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 97f490f09d0a47788bd8bfb1c6d1be34: Sending DeleteTablet for 3 replicas of tablet 27ae008ddbe649ec84824d081ce78c3e
I20260203 21:07:03.617655 5767 tablet_service.cc:1552] Processing DeleteTablet for tablet 27ae008ddbe649ec84824d081ce78c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:03 UTC) from {username='slave'} at 127.0.0.1:53104
I20260203 21:07:03.617658 5898 tablet_service.cc:1552] Processing DeleteTablet for tablet 27ae008ddbe649ec84824d081ce78c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:03 UTC) from {username='slave'} at 127.0.0.1:50358
I20260203 21:07:03.618072 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 5705
I20260203 21:07:03.621795 6029 tablet_service.cc:1552] Processing DeleteTablet for tablet 27ae008ddbe649ec84824d081ce78c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:03 UTC) from {username='slave'} at 127.0.0.1:42980
I20260203 21:07:03.622121 6136 tablet_replica.cc:333] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d: stopping tablet replica
I20260203 21:07:03.622367 6137 tablet_replica.cc:333] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b: stopping tablet replica
I20260203 21:07:03.622428 6136 raft_consensus.cc:2243] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:03.622697 6137 raft_consensus.cc:2243] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [term 1 LEADER]: Raft consensus shutting down.
I20260203 21:07:03.622856 6136 raft_consensus.cc:2272] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:03.623162 6137 raft_consensus.cc:2272] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:03.624471 6136 ts_tablet_manager.cc:1916] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:03.624850 6137 ts_tablet_manager.cc:1916] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260203 21:07:03.626873 5631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.102.129:36243 (error 108)
I20260203 21:07:03.627344 6136 ts_tablet_manager.cc:1929] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:07:03.627492 6136 log.cc:1199] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-1/wal/wals/27ae008ddbe649ec84824d081ce78c3e
I20260203 21:07:03.627565 6137 ts_tablet_manager.cc:1929] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:07:03.627696 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 5836
I20260203 21:07:03.627864 6136 ts_tablet_manager.cc:1950] T 27ae008ddbe649ec84824d081ce78c3e P 932b6455e731419fb5747ab9d59aca8d: Deleting consensus metadata
I20260203 21:07:03.627756 6137 log.cc:1199] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1770152788143875-410-0/minicluster-data/ts-2/wal/wals/27ae008ddbe649ec84824d081ce78c3e
I20260203 21:07:03.629333 6137 ts_tablet_manager.cc:1950] T 27ae008ddbe649ec84824d081ce78c3e P 2e9c1e9456914a6f88d8b29e272b529b: Deleting consensus metadata
I20260203 21:07:03.630457 5632 catalog_manager.cc:4985] TS 2e9c1e9456914a6f88d8b29e272b529b (127.0.102.131:46377): tablet 27ae008ddbe649ec84824d081ce78c3e (table test-table [id=1b07bfe6c10446fcb45252cc3793dacf]) successfully deleted
I20260203 21:07:03.630695 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.628919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:36243 (local address 127.0.0.1:53114)
0203 21:07:03.630523 (+ 1604us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:03.630596 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.129:36243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1505}
W20260203 21:07:03.630903 5631 catalog_manager.cc:4712] TS a55a05657b8d406280b32938b4b1d8ea (127.0.102.129:36243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27ae008ddbe649ec84824d081ce78c3e: Network error: Client connection negotiation failed: client connection to 127.0.102.129:36243: connect: Connection refused (error 111)
I20260203 21:07:03.633901 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 5967
I20260203 21:07:03.634872 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.634117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:37355 (local address 127.0.0.1:50364)
0203 21:07:03.634744 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:03.634792 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.130:37355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542}
I20260203 21:07:03.638715 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 5614
2026-02-03T21:07:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (896 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260203 21:07:03.658663 410 mini_oidc.cc:150] Starting JWKS server
I20260203 21:07:03.660045 410 webserver.cc:492] Webserver started at https://127.0.0.1:36137/ using document root <none> and password file <none>
I20260203 21:07:03.660161 410 mini_oidc.cc:186] Starting OIDC Discovery server
I20260203 21:07:03.660343 410 webserver.cc:492] Webserver started at http://127.0.0.1:42953/ using document root <none> and password file <none>
2026-02-03T21:07:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:07:03Z Disabled control of system clock
I20260203 21:07:03.667309 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:35167
--webserver_interface=127.0.102.190
--webserver_port=0
--builtin_ntp_servers=127.0.102.148:38025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:35167
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36137/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/testchainca.cert with env {}
W20260203 21:07:03.766625 6144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:03.766834 6144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:03.766901 6144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:03.770216 6144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:07:03.770275 6144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:03.770298 6144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:03.770334 6144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:07:03.770355 6144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260203 21:07:03.770378 6144 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260203 21:07:03.770422 6144 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36137/jwks.json
I20260203 21:07:03.776269 6144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:38025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:35167
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:35167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36137/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:03.777522 6144 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:03.778455 6144 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:03.784094 6150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:03.784129 6149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:03.784137 6152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:03.784233 6144 server_base.cc:1060] running on GCE node
I20260203 21:07:03.784793 6144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:03.785570 6144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:03.786721 6144 hybrid_clock.cc:648] HybridClock initialized: now 1770152823786704 us; error 30 us; skew 500 ppm
I20260203 21:07:03.788336 6144 webserver.cc:492] Webserver started at http://127.0.102.190:33835/ using document root <none> and password file <none>
I20260203 21:07:03.788898 6144 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:03.788951 6144 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:03.789125 6144 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:03.790704 6144 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "d73416bcaef64b42bdf91fc41187bc39"
format_stamp: "Formatted at 2026-02-03 21:07:03 on dist-test-slave-j889"
server_key: "443b6bff9b4476a45f4795552bd4f925"
server_key_iv: "e68f8310788f835fd24f3abcdc7c5ac5"
server_key_version: "encryptionkey@0"
I20260203 21:07:03.791167 6144 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "d73416bcaef64b42bdf91fc41187bc39"
format_stamp: "Formatted at 2026-02-03 21:07:03 on dist-test-slave-j889"
server_key: "443b6bff9b4476a45f4795552bd4f925"
server_key_iv: "e68f8310788f835fd24f3abcdc7c5ac5"
server_key_version: "encryptionkey@0"
I20260203 21:07:03.794207 6144 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260203 21:07:03.796126 6158 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:03.797079 6144 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:03.797174 6144 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "d73416bcaef64b42bdf91fc41187bc39"
format_stamp: "Formatted at 2026-02-03 21:07:03 on dist-test-slave-j889"
server_key: "443b6bff9b4476a45f4795552bd4f925"
server_key_iv: "e68f8310788f835fd24f3abcdc7c5ac5"
server_key_version: "encryptionkey@0"
I20260203 21:07:03.797250 6144 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:03.810783 6144 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:03.811329 6144 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:03.811477 6144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:03.817869 6210 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:35167 every 8 connection(s)
I20260203 21:07:03.817878 6144 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:35167
I20260203 21:07:03.818845 6144 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:07:03.821354 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 6144
I20260203 21:07:03.821460 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:07:03.821458 6211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:03.821676 410 external_mini_cluster.cc:1468] Setting key 6e1141d5b16e5c8e756dbf7f01fed30f
I20260203 21:07:03.827080 6211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39: Bootstrap starting.
I20260203 21:07:03.829041 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.822694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.0.102.190:35167)
0203 21:07:03.823243 (+ 549us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:03.823250 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:03.823281 (+ 31us) server_negotiation.cc:408] Connection header received
0203 21:07:03.824399 (+ 1118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:03.824417 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:03.824780 (+ 363us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:03.825056 (+ 276us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:03.825508 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:03.826247 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:03.826895 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:03.827151 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:03.827585 (+ 434us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:03.827627 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:03.827636 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:03.827989 (+ 353us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:03.828026 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:03.828041 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:03.828127 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:03.828269 (+ 142us) server_negotiation.cc:300] Negotiation successful
0203 21:07:03.828467 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260203 21:07:03.829178 6211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:03.829824 6211 log.cc:826] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:03.831385 410 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260203 21:07:03.831578 6211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39: No bootstrap required, opened a new log
I20260203 21:07:03.834005 6211 raft_consensus.cc:359] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73416bcaef64b42bdf91fc41187bc39" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 35167 } }
I20260203 21:07:03.834170 6211 raft_consensus.cc:385] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:03.834230 6211 raft_consensus.cc:740] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73416bcaef64b42bdf91fc41187bc39, State: Initialized, Role: FOLLOWER
I20260203 21:07:03.834599 6211 consensus_queue.cc:260] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73416bcaef64b42bdf91fc41187bc39" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 35167 } }
I20260203 21:07:03.834721 6211 raft_consensus.cc:399] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:07:03.834774 6211 raft_consensus.cc:493] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:07:03.834857 6211 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:03.835577 6211 raft_consensus.cc:515] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73416bcaef64b42bdf91fc41187bc39" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 35167 } }
I20260203 21:07:03.835870 6211 leader_election.cc:304] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d73416bcaef64b42bdf91fc41187bc39; no voters:
I20260203 21:07:03.836124 6211 leader_election.cc:290] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:07:03.836196 6216 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:03.836351 6216 raft_consensus.cc:697] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [term 1 LEADER]: Becoming Leader. State: Replica: d73416bcaef64b42bdf91fc41187bc39, State: Running, Role: LEADER
I20260203 21:07:03.836613 6216 consensus_queue.cc:237] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73416bcaef64b42bdf91fc41187bc39" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 35167 } }
I20260203 21:07:03.837204 6211 sys_catalog.cc:565] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:07:03.838323 6218 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d73416bcaef64b42bdf91fc41187bc39. Latest consensus state: current_term: 1 leader_uuid: "d73416bcaef64b42bdf91fc41187bc39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73416bcaef64b42bdf91fc41187bc39" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 35167 } } }
I20260203 21:07:03.838431 6218 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:03.838517 6217 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d73416bcaef64b42bdf91fc41187bc39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73416bcaef64b42bdf91fc41187bc39" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 35167 } } }
I20260203 21:07:03.838598 6217 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39 [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:03.839375 6225 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:07:03.841862 6225 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:07:03.846489 6225 catalog_manager.cc:1357] Generated new cluster ID: e61dc4025293448897a146e618f82f93
I20260203 21:07:03.846547 6225 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:07:03.852281 6225 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:07:03.853125 6225 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:07:03.857941 6225 catalog_manager.cc:6027] T 00000000000000000000000000000000 P d73416bcaef64b42bdf91fc41187bc39: Generated new TSK 0
I20260203 21:07:03.858525 6225 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260203 21:07:03.901775 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.870361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.0.102.190:35167)
0203 21:07:03.870504 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:03.870509 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:03.870598 (+ 89us) server_negotiation.cc:408] Connection header received
0203 21:07:03.870785 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:03.870789 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:03.870847 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:03.870953 (+ 106us) server_negotiation.cc:227] Negotiated authn=JWT
0203 21:07:03.871293 (+ 340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:03.872102 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:03.872819 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:03.873019 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:03.873124 (+ 105us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0203 21:07:03.901294 (+ 28170us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0203 21:07:03.901400 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:03.901579 (+ 179us) server_negotiation.cc:300] Negotiation successful
0203 21:07:03.901630 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54,"thread_start_us":118,"threads_started":1}
I20260203 21:07:03.910959 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:03.907422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.0.102.190:35167)
0203 21:07:03.907550 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:03.907553 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:03.907699 (+ 146us) server_negotiation.cc:408] Connection header received
0203 21:07:03.907824 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:03.907827 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:03.907866 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:03.907924 (+ 58us) server_negotiation.cc:227] Negotiated authn=JWT
0203 21:07:03.908321 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:03.908920 (+ 599us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:03.909777 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:03.909910 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:03.909948 (+ 38us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0203 21:07:03.910700 (+ 752us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0203 21:07:03.910871 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54360: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":44}
W20260203 21:07:03.911020 6214 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54360: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20260203 21:07:03.911820 6247 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:03.907355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:35167 (local address 127.0.0.1:54360)
0203 21:07:03.907624 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:03.907639 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:03.907741 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:03.907946 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:03.907950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:03.907957 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0203 21:07:03.908234 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:03.908242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:03.909037 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:03.909042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:03.909682 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:03.909692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:03.909800 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:03.909823 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0203 21:07:03.910875 (+ 1052us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0203 21:07:03.911030 (+ 155us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.102.190:35167: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
I20260203 21:07:18.919157 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:18.915923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.0.102.190:35167)
0203 21:07:18.916195 (+ 272us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:18.916200 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:18.916214 (+ 14us) server_negotiation.cc:408] Connection header received
0203 21:07:18.916267 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:18.916269 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:18.916313 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:18.916387 (+ 74us) server_negotiation.cc:227] Negotiated authn=JWT
0203 21:07:18.916772 (+ 385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:18.917302 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:18.917983 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:18.918173 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:18.918220 (+ 47us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0203 21:07:18.918852 (+ 632us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0203 21:07:18.919012 (+ 160us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41414: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
W20260203 21:07:18.919224 6252 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:18.915748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:35167 (local address 127.0.0.1:41414)
0203 21:07:18.916015 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:18.916029 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:18.916141 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:18.916410 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:18.916413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:18.916421 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0203 21:07:18.916652 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:18.916662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:18.917423 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:18.917428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:18.917867 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:18.917876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:18.917982 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:18.918007 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0203 21:07:18.918962 (+ 955us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0203 21:07:18.919100 (+ 138us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.102.190:35167: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":113,"threads_started":1}
W20260203 21:07:18.919237 6253 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41414: JWT verification failed: token expired
W20260203 21:07:18.921581 6253 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:18.920896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.0.102.190:35167)
0203 21:07:18.921012 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:18.921016 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:18.921117 (+ 101us) server_negotiation.cc:408] Connection header received
0203 21:07:18.921232 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:18.921235 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:18.921282 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:18.921342 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:18.921495 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41416: BlockingRecv error: recv got EOF from 127.0.0.1:41416 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260203 21:07:18.925025 6263 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.0.102.190:35167
W20260203 21:07:18.925612 6253 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:18.924771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.0.102.190:35167)
0203 21:07:18.924901 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:18.924905 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:18.924986 (+ 81us) server_negotiation.cc:408] Connection header received
0203 21:07:18.925210 (+ 224us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:18.925213 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:18.925260 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:18.925332 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:18.925507 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41430: BlockingRecv error: recv got EOF from 127.0.0.1:41430 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260203 21:07:18.925935 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 6144
2026-02-03T21:07:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15306 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260203 21:07:18.965395 410 mini_oidc.cc:150] Starting JWKS server
I20260203 21:07:18.966300 410 webserver.cc:492] Webserver started at https://127.0.0.1:39269/ using document root <none> and password file <none>
I20260203 21:07:18.966389 410 mini_oidc.cc:186] Starting OIDC Discovery server
I20260203 21:07:18.966619 410 webserver.cc:492] Webserver started at http://127.0.0.1:45553/ using document root <none> and password file <none>
2026-02-03T21:07:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:07:18Z Disabled control of system clock
I20260203 21:07:18.973999 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:46507
--webserver_interface=127.0.102.190
--webserver_port=0
--builtin_ntp_servers=127.0.102.148:43911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:46507
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39269/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/testchainca.cert with env {}
W20260203 21:07:19.078657 6269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:19.078907 6269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:19.078959 6269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:19.082360 6269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:07:19.082435 6269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:19.082451 6269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:19.082469 6269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:07:19.082485 6269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260203 21:07:19.082501 6269 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260203 21:07:19.082515 6269 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39269/jwks.json
I20260203 21:07:19.087042 6269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:43911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:46507
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:46507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39269/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:19.088153 6269 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:19.088972 6269 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:19.094444 6274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:19.094596 6275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:19.094697 6269 server_base.cc:1060] running on GCE node
W20260203 21:07:19.094594 6277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:19.095252 6269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:19.096151 6269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:19.097288 6269 hybrid_clock.cc:648] HybridClock initialized: now 1770152839097274 us; error 33 us; skew 500 ppm
I20260203 21:07:19.099045 6269 webserver.cc:492] Webserver started at http://127.0.102.190:46471/ using document root <none> and password file <none>
I20260203 21:07:19.099609 6269 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:19.099668 6269 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:19.099854 6269 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:19.101552 6269 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "e1a44c93553846fa8861f12870f2e2d4"
format_stamp: "Formatted at 2026-02-03 21:07:19 on dist-test-slave-j889"
server_key: "f38b88826fc9a8e819fd521947fe56d9"
server_key_iv: "e2da7066ae7340de87493cbeffd12363"
server_key_version: "encryptionkey@0"
I20260203 21:07:19.102011 6269 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "e1a44c93553846fa8861f12870f2e2d4"
format_stamp: "Formatted at 2026-02-03 21:07:19 on dist-test-slave-j889"
server_key: "f38b88826fc9a8e819fd521947fe56d9"
server_key_iv: "e2da7066ae7340de87493cbeffd12363"
server_key_version: "encryptionkey@0"
I20260203 21:07:19.105237 6269 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260203 21:07:19.107335 6283 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:19.108443 6269 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:19.108561 6269 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "e1a44c93553846fa8861f12870f2e2d4"
format_stamp: "Formatted at 2026-02-03 21:07:19 on dist-test-slave-j889"
server_key: "f38b88826fc9a8e819fd521947fe56d9"
server_key_iv: "e2da7066ae7340de87493cbeffd12363"
server_key_version: "encryptionkey@0"
I20260203 21:07:19.108692 6269 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:19.124089 6269 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:19.124823 6269 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:19.124987 6269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:19.132074 6335 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:46507 every 8 connection(s)
I20260203 21:07:19.132081 6269 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:46507
I20260203 21:07:19.133153 6269 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:07:19.135980 6336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:19.138623 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 6269
I20260203 21:07:19.138723 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:07:19.138932 410 external_mini_cluster.cc:1468] Setting key d9a1a2a845e382c233d778336dd47cf3
I20260203 21:07:19.141557 6336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4: Bootstrap starting.
I20260203 21:07:19.143575 6336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:19.144290 6336 log.cc:826] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:19.145998 6336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4: No bootstrap required, opened a new log
I20260203 21:07:19.148458 6336 raft_consensus.cc:359] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a44c93553846fa8861f12870f2e2d4" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46507 } }
I20260203 21:07:19.148672 6336 raft_consensus.cc:385] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:19.148754 6336 raft_consensus.cc:740] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a44c93553846fa8861f12870f2e2d4, State: Initialized, Role: FOLLOWER
I20260203 21:07:19.149202 6336 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a44c93553846fa8861f12870f2e2d4" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46507 } }
I20260203 21:07:19.149329 6336 raft_consensus.cc:399] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:07:19.149387 6336 raft_consensus.cc:493] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:07:19.149475 6336 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:19.149504 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:19.140071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.0.102.190:46507)
0203 21:07:19.140726 (+ 655us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:19.140735 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:19.140769 (+ 34us) server_negotiation.cc:408] Connection header received
0203 21:07:19.141316 (+ 547us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:19.141334 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:19.141693 (+ 359us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:19.141988 (+ 295us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:19.142437 (+ 449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.143185 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:19.143846 (+ 661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.144118 (+ 272us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:19.147877 (+ 3759us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:19.147899 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:19.147910 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:19.148293 (+ 383us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:19.148337 (+ 44us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:19.148352 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:19.148451 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:19.148604 (+ 153us) server_negotiation.cc:300] Negotiation successful
0203 21:07:19.148892 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":124,"threads_started":1}
I20260203 21:07:19.150503 6336 raft_consensus.cc:515] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a44c93553846fa8861f12870f2e2d4" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46507 } }
I20260203 21:07:19.150838 6336 leader_election.cc:304] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1a44c93553846fa8861f12870f2e2d4; no voters:
I20260203 21:07:19.151091 6336 leader_election.cc:290] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:07:19.151196 6341 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:19.151396 6341 raft_consensus.cc:697] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [term 1 LEADER]: Becoming Leader. State: Replica: e1a44c93553846fa8861f12870f2e2d4, State: Running, Role: LEADER
I20260203 21:07:19.151739 6341 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a44c93553846fa8861f12870f2e2d4" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46507 } }
I20260203 21:07:19.152205 6336 sys_catalog.cc:565] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:07:19.152457 410 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260203 21:07:19.153317 6342 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1a44c93553846fa8861f12870f2e2d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a44c93553846fa8861f12870f2e2d4" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46507 } } }
I20260203 21:07:19.153429 6342 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:19.154256 6343 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1a44c93553846fa8861f12870f2e2d4. Latest consensus state: current_term: 1 leader_uuid: "e1a44c93553846fa8861f12870f2e2d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a44c93553846fa8861f12870f2e2d4" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46507 } } }
I20260203 21:07:19.154405 6343 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4 [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:19.154510 6350 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:07:19.157423 6350 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:07:19.162817 6350 catalog_manager.cc:1357] Generated new cluster ID: af5de482d0f14dc3a2799b62040df948
I20260203 21:07:19.162899 6350 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:07:19.187811 6350 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:07:19.188829 6350 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:07:19.197146 6350 catalog_manager.cc:6027] T 00000000000000000000000000000000 P e1a44c93553846fa8861f12870f2e2d4: Generated new TSK 0
I20260203 21:07:19.197890 6350 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260203 21:07:19.223034 6366 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260203 21:07:19.223114 6366 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260203 21:07:19.223865 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:19.207394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.0.102.190:46507)
0203 21:07:19.207535 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:19.207540 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:19.207676 (+ 136us) server_negotiation.cc:408] Connection header received
0203 21:07:19.207887 (+ 211us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:19.207891 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:19.207957 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:19.208056 (+ 99us) server_negotiation.cc:227] Negotiated authn=JWT
0203 21:07:19.208406 (+ 350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.209272 (+ 866us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:19.209962 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.210168 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:19.210278 (+ 110us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0203 21:07:19.223428 (+ 13150us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0203 21:07:19.223721 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56598: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":59}
W20260203 21:07:19.223942 6339 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56598: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260203 21:07:19.223985 6365 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:19.207277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46507 (local address 127.0.0.1:56598)
0203 21:07:19.207591 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:19.207605 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:19.207716 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:19.208068 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:19.208071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:19.208077 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0203 21:07:19.208313 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:19.208320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.209428 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:19.209432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:19.209860 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:19.209869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.209967 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:19.209988 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0203 21:07:19.223657 (+ 13669us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0203 21:07:19.223811 (+ 154us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.102.190:46507: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":114,"threads_started":1}
I20260203 21:07:19.224750 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 6269
2026-02-03T21:07:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (297 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260203 21:07:19.263182 410 mini_oidc.cc:150] Starting JWKS server
I20260203 21:07:19.264021 410 webserver.cc:492] Webserver started at https://127.0.0.1:34531/ using document root <none> and password file <none>
I20260203 21:07:19.264102 410 mini_oidc.cc:186] Starting OIDC Discovery server
I20260203 21:07:19.264289 410 webserver.cc:492] Webserver started at http://127.0.0.1:44795/ using document root <none> and password file <none>
2026-02-03T21:07:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:07:19Z Disabled control of system clock
I20260203 21:07:19.271274 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:43225
--webserver_interface=127.0.102.190
--webserver_port=0
--builtin_ntp_servers=127.0.102.148:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:43225
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34531/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260203 21:07:19.374740 6372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:19.374966 6372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:19.375036 6372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:19.378304 6372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:07:19.378363 6372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:19.378386 6372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:19.378412 6372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:07:19.378443 6372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260203 21:07:19.378473 6372 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260203 21:07:19.378501 6372 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34531/jwks.json
I20260203 21:07:19.382345 6372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:43225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:43225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34531/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:19.383358 6372 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:19.384469 6372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:19.389714 6378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:19.389756 6377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:19.389714 6380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:19.390141 6372 server_base.cc:1060] running on GCE node
I20260203 21:07:19.390691 6372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:19.391711 6372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:19.392872 6372 hybrid_clock.cc:648] HybridClock initialized: now 1770152839392850 us; error 38 us; skew 500 ppm
I20260203 21:07:19.394683 6372 webserver.cc:492] Webserver started at http://127.0.102.190:44789/ using document root <none> and password file <none>
I20260203 21:07:19.395258 6372 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:19.395313 6372 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:19.395484 6372 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:19.397203 6372 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "6e5690a53d134ffa844995282aa21286"
format_stamp: "Formatted at 2026-02-03 21:07:19 on dist-test-slave-j889"
server_key: "54e226846c2c237fb1dbbce3639fda7c"
server_key_iv: "3a35b1dd43ff8a7b23f30c8156e087d1"
server_key_version: "encryptionkey@0"
I20260203 21:07:19.397670 6372 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "6e5690a53d134ffa844995282aa21286"
format_stamp: "Formatted at 2026-02-03 21:07:19 on dist-test-slave-j889"
server_key: "54e226846c2c237fb1dbbce3639fda7c"
server_key_iv: "3a35b1dd43ff8a7b23f30c8156e087d1"
server_key_version: "encryptionkey@0"
I20260203 21:07:19.400930 6372 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260203 21:07:19.403235 6386 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:19.404392 6372 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:19.404526 6372 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "6e5690a53d134ffa844995282aa21286"
format_stamp: "Formatted at 2026-02-03 21:07:19 on dist-test-slave-j889"
server_key: "54e226846c2c237fb1dbbce3639fda7c"
server_key_iv: "3a35b1dd43ff8a7b23f30c8156e087d1"
server_key_version: "encryptionkey@0"
I20260203 21:07:19.404688 6372 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:19.417979 6372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:19.418638 6372 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:19.418807 6372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:19.425510 6372 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:43225
I20260203 21:07:19.425541 6438 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:43225 every 8 connection(s)
I20260203 21:07:19.426535 6372 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:07:19.429474 6439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:19.435227 6439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286: Bootstrap starting.
I20260203 21:07:19.435932 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 6372
I20260203 21:07:19.436038 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:07:19.436249 410 external_mini_cluster.cc:1468] Setting key 7ec80cae460609559bf196c949b5f056
I20260203 21:07:19.437940 6439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:19.438687 6439 log.cc:826] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:19.440829 6439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286: No bootstrap required, opened a new log
I20260203 21:07:19.443998 6439 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5690a53d134ffa844995282aa21286" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43225 } }
I20260203 21:07:19.444235 6439 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:19.444308 6439 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5690a53d134ffa844995282aa21286, State: Initialized, Role: FOLLOWER
I20260203 21:07:19.444434 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:19.437469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.0.102.190:43225)
0203 21:07:19.438144 (+ 675us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:19.438154 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:19.438188 (+ 34us) server_negotiation.cc:408] Connection header received
0203 21:07:19.438750 (+ 562us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:19.438767 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:19.439158 (+ 391us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:19.439459 (+ 301us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:19.439943 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.441047 (+ 1104us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:19.441767 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.442047 (+ 280us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:19.442555 (+ 508us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:19.442578 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:19.442591 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:19.443085 (+ 494us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:19.443143 (+ 58us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:19.443161 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:19.443268 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:19.443419 (+ 151us) server_negotiation.cc:300] Negotiation successful
0203 21:07:19.443678 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":145,"threads_started":1}
I20260203 21:07:19.445513 6439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5690a53d134ffa844995282aa21286" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43225 } }
I20260203 21:07:19.445660 6439 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:07:19.445721 6439 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:07:19.445811 6439 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:19.446784 6439 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5690a53d134ffa844995282aa21286" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43225 } }
I20260203 21:07:19.447149 410 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260203 21:07:19.447144 6439 leader_election.cc:304] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e5690a53d134ffa844995282aa21286; no voters:
I20260203 21:07:19.447434 6439 leader_election.cc:290] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:07:19.447546 6444 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:19.447742 6444 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [term 1 LEADER]: Becoming Leader. State: Replica: 6e5690a53d134ffa844995282aa21286, State: Running, Role: LEADER
I20260203 21:07:19.448330 6444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5690a53d134ffa844995282aa21286" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43225 } }
I20260203 21:07:19.448563 6439 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:07:19.449951 6446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e5690a53d134ffa844995282aa21286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5690a53d134ffa844995282aa21286" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43225 } } }
I20260203 21:07:19.450065 6446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:19.450043 6445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e5690a53d134ffa844995282aa21286. Latest consensus state: current_term: 1 leader_uuid: "6e5690a53d134ffa844995282aa21286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5690a53d134ffa844995282aa21286" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43225 } } }
I20260203 21:07:19.450127 6445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286 [sys.catalog]: This master's current role is: LEADER
W20260203 21:07:19.453265 6459 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260203 21:07:19.453330 6459 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260203 21:07:19.453400 6454 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:07:19.454085 6454 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:07:19.458727 6454 catalog_manager.cc:1357] Generated new cluster ID: 7f845a6584a84046a76cf7e203312245
I20260203 21:07:19.458789 6454 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:07:19.466253 6454 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:07:19.467494 6454 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:07:19.479103 6454 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 6e5690a53d134ffa844995282aa21286: Generated new TSK 0
I20260203 21:07:19.479921 6454 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260203 21:07:19.511221 6469 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260203 21:07:19.511289 6469 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260203 21:07:19.512777 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:19.494246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.0.102.190:43225)
0203 21:07:19.494372 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:19.494382 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:19.494518 (+ 136us) server_negotiation.cc:408] Connection header received
0203 21:07:19.494699 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:19.494703 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:19.494765 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:19.494876 (+ 111us) server_negotiation.cc:227] Negotiated authn=JWT
0203 21:07:19.495181 (+ 305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.495992 (+ 811us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:19.496738 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.496952 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:19.497065 (+ 113us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0203 21:07:19.512299 (+ 15234us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0203 21:07:19.512610 (+ 311us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41844: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":36}
W20260203 21:07:19.512835 6468 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:19.494129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:43225 (local address 127.0.0.1:41844)
0203 21:07:19.494445 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:19.494457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:19.494565 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:19.494875 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:19.494878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:19.494884 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0203 21:07:19.495087 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:19.495093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.496138 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:19.496142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:19.496569 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:19.496578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.496745 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:19.496769 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0203 21:07:19.512498 (+ 15729us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0203 21:07:19.512686 (+ 188us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.102.190:43225: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":116,"threads_started":1}
W20260203 21:07:19.512840 6442 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41844: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260203 21:07:19.513576 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 6372
2026-02-03T21:07:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (288 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-02-03T21:07:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:07:19Z Disabled control of system clock
I20260203 21:07:19.559020 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:39867
--webserver_interface=127.0.102.190
--webserver_port=0
--builtin_ntp_servers=127.0.102.148:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:39867
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/testchainca.cert with env {}
W20260203 21:07:19.662504 6473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:19.662737 6473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:19.662809 6473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:19.666188 6473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:07:19.666271 6473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:19.666294 6473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:19.666318 6473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:07:19.666347 6473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260203 21:07:19.666379 6473 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260203 21:07:19.666401 6473 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260203 21:07:19.670872 6473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:39867
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:39867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:19.672008 6473 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:19.672911 6473 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:19.678341 6479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:19.678447 6478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:19.678359 6481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:19.678624 6473 server_base.cc:1060] running on GCE node
I20260203 21:07:19.679198 6473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:19.680064 6473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:19.681232 6473 hybrid_clock.cc:648] HybridClock initialized: now 1770152839681204 us; error 39 us; skew 500 ppm
I20260203 21:07:19.683050 6473 webserver.cc:492] Webserver started at http://127.0.102.190:42691/ using document root <none> and password file <none>
I20260203 21:07:19.683605 6473 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:19.683660 6473 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:19.683830 6473 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:19.685606 6473 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "cfed42af5f824160b9a5d1dff6973856"
format_stamp: "Formatted at 2026-02-03 21:07:19 on dist-test-slave-j889"
server_key: "af462fb045c5e1f751b96aa31e610cd7"
server_key_iv: "0884f527eef4b809c289040ff719d8ab"
server_key_version: "encryptionkey@0"
I20260203 21:07:19.686074 6473 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "cfed42af5f824160b9a5d1dff6973856"
format_stamp: "Formatted at 2026-02-03 21:07:19 on dist-test-slave-j889"
server_key: "af462fb045c5e1f751b96aa31e610cd7"
server_key_iv: "0884f527eef4b809c289040ff719d8ab"
server_key_version: "encryptionkey@0"
I20260203 21:07:19.689502 6473 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260203 21:07:19.691829 6487 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:19.692974 6473 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:07:19.693084 6473 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "cfed42af5f824160b9a5d1dff6973856"
format_stamp: "Formatted at 2026-02-03 21:07:19 on dist-test-slave-j889"
server_key: "af462fb045c5e1f751b96aa31e610cd7"
server_key_iv: "0884f527eef4b809c289040ff719d8ab"
server_key_version: "encryptionkey@0"
I20260203 21:07:19.693174 6473 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:19.733520 6473 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:19.734256 6473 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:19.734424 6473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:19.741304 6539 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:39867 every 8 connection(s)
I20260203 21:07:19.741309 6473 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:39867
I20260203 21:07:19.742435 6473 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:07:19.744182 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 6473
I20260203 21:07:19.744307 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:07:19.744553 410 external_mini_cluster.cc:1468] Setting key 856c059a6fefcbdd7b934089344b26fd
I20260203 21:07:19.745541 6540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:19.750928 6540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856: Bootstrap starting.
I20260203 21:07:19.751807 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:19.745743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.0.102.190:39867)
0203 21:07:19.746322 (+ 579us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:19.746331 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:19.746364 (+ 33us) server_negotiation.cc:408] Connection header received
0203 21:07:19.746992 (+ 628us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:19.747009 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:19.747325 (+ 316us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:19.747619 (+ 294us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:19.748060 (+ 441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.748862 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:19.749540 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.749833 (+ 293us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:19.750287 (+ 454us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:19.750328 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:19.750337 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:19.750715 (+ 378us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:19.750754 (+ 39us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:19.750768 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:19.750857 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:19.751010 (+ 153us) server_negotiation.cc:300] Negotiation successful
0203 21:07:19.751251 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":115,"threads_started":1}
I20260203 21:07:19.753165 6540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:19.753897 6540 log.cc:826] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:19.754484 410 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260203 21:07:19.755877 6540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856: No bootstrap required, opened a new log
I20260203 21:07:19.758355 6540 raft_consensus.cc:359] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfed42af5f824160b9a5d1dff6973856" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 39867 } }
I20260203 21:07:19.758541 6540 raft_consensus.cc:385] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:19.758587 6540 raft_consensus.cc:740] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfed42af5f824160b9a5d1dff6973856, State: Initialized, Role: FOLLOWER
I20260203 21:07:19.759030 6540 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfed42af5f824160b9a5d1dff6973856" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 39867 } }
I20260203 21:07:19.759158 6540 raft_consensus.cc:399] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:07:19.759202 6540 raft_consensus.cc:493] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:07:19.759267 6540 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:19.760020 6540 raft_consensus.cc:515] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfed42af5f824160b9a5d1dff6973856" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 39867 } }
I20260203 21:07:19.760317 6540 leader_election.cc:304] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfed42af5f824160b9a5d1dff6973856; no voters:
I20260203 21:07:19.760579 6540 leader_election.cc:290] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:07:19.760773 6545 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:19.760962 6545 raft_consensus.cc:697] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [term 1 LEADER]: Becoming Leader. State: Replica: cfed42af5f824160b9a5d1dff6973856, State: Running, Role: LEADER
I20260203 21:07:19.761245 6545 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfed42af5f824160b9a5d1dff6973856" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 39867 } }
I20260203 21:07:19.761693 6540 sys_catalog.cc:565] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:07:19.762817 6546 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfed42af5f824160b9a5d1dff6973856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfed42af5f824160b9a5d1dff6973856" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 39867 } } }
I20260203 21:07:19.762926 6546 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:19.763293 6554 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:07:19.763373 6547 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfed42af5f824160b9a5d1dff6973856. Latest consensus state: current_term: 1 leader_uuid: "cfed42af5f824160b9a5d1dff6973856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfed42af5f824160b9a5d1dff6973856" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 39867 } } }
I20260203 21:07:19.763898 6547 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856 [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:19.766753 6554 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:07:19.772215 6554 catalog_manager.cc:1357] Generated new cluster ID: 7201f96b48364638957e99fef4c6112f
I20260203 21:07:19.772308 6554 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:07:19.788722 6554 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:07:19.789677 6554 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:07:19.796975 6554 catalog_manager.cc:6027] T 00000000000000000000000000000000 P cfed42af5f824160b9a5d1dff6973856: Generated new TSK 0
I20260203 21:07:19.797958 6554 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260203 21:07:19.864233 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:19.814838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.0.102.190:39867)
0203 21:07:19.814978 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:19.814982 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:19.815074 (+ 92us) server_negotiation.cc:408] Connection header received
0203 21:07:19.815288 (+ 214us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:19.815291 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:19.815347 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:19.815444 (+ 97us) server_negotiation.cc:227] Negotiated authn=JWT
0203 21:07:19.815852 (+ 408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.816364 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:19.817300 (+ 936us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.817469 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:19.817568 (+ 99us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0203 21:07:19.863839 (+ 46271us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0203 21:07:19.864114 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37408: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260203 21:07:19.864305 6543 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37408: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20260203 21:07:19.864449 6569 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:19.814695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:39867 (local address 127.0.0.1:37408)
0203 21:07:19.814986 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:19.815001 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:19.815116 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:19.815457 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:19.815460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:19.815467 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0203 21:07:19.815752 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:19.815761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.816475 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:19.816479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:19.817174 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:19.817186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:19.817300 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:19.817325 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0203 21:07:19.864072 (+ 46747us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0203 21:07:19.864252 (+ 180us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.102.190:39867: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":131,"threads_started":1}
I20260203 21:07:19.865015 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 6473
2026-02-03T21:07:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (330 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (115 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (115 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:07:20 dist-test-slave-j889 krb5kdc[6581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:07:20 dist-test-slave-j889 krb5kdc[6581](info): set up 2 sockets
Feb 03 21:07:20 dist-test-slave-j889 krb5kdc[6581](info): commencing operation
krb5kdc: starting...
W20260203 21:07:21.937151 410 mini_kdc.cc:121] Time spent starting KDC: real 1.823s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:07:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:07:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:07:22.059536 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:36633
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:36633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:22.160012 6597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:22.160231 6597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:22.160285 6597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:22.163422 6597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:07:22.163475 6597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:22.163496 6597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:22.163513 6597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:07:22.163532 6597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:07:22.167663 6597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:36633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:36633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:22.168622 6597 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:22.169409 6597 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:22.174652 6605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:22.174656 6602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:22.174654 6603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:22.175043 6597 server_base.cc:1060] running on GCE node
I20260203 21:07:22.175529 6597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:22.176396 6597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:22.177548 6597 hybrid_clock.cc:648] HybridClock initialized: now 1770152842177525 us; error 46 us; skew 500 ppm
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:22.179875 6597 init.cc:377] Logged in from keytab as kudu/127.0.102.190@KRBTEST.COM (short username kudu)
I20260203 21:07:22.180879 6597 webserver.cc:492] Webserver started at http://127.0.102.190:40911/ using document root <none> and password file <none>
I20260203 21:07:22.181384 6597 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:22.181433 6597 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:22.181600 6597 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:22.183194 6597 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "37d568b8599c4dc490c48b88c56f295e"
format_stamp: "Formatted at 2026-02-03 21:07:22 on dist-test-slave-j889"
server_key: "003547dc33445f8da02a21ec2776b014"
server_key_iv: "82d621074e93a93dccedddb8a5918990"
server_key_version: "encryptionkey@0"
I20260203 21:07:22.183619 6597 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "37d568b8599c4dc490c48b88c56f295e"
format_stamp: "Formatted at 2026-02-03 21:07:22 on dist-test-slave-j889"
server_key: "003547dc33445f8da02a21ec2776b014"
server_key_iv: "82d621074e93a93dccedddb8a5918990"
server_key_version: "encryptionkey@0"
I20260203 21:07:22.186738 6597 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260203 21:07:22.188807 6612 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:22.189733 6597 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:22.189831 6597 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "37d568b8599c4dc490c48b88c56f295e"
format_stamp: "Formatted at 2026-02-03 21:07:22 on dist-test-slave-j889"
server_key: "003547dc33445f8da02a21ec2776b014"
server_key_iv: "82d621074e93a93dccedddb8a5918990"
server_key_version: "encryptionkey@0"
I20260203 21:07:22.189934 6597 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:22.215692 6597 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:22.218412 6597 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:22.218571 6597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:22.225100 6597 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:36633
I20260203 21:07:22.225104 6664 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:36633 every 8 connection(s)
I20260203 21:07:22.226044 6597 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:07:22.228652 6665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:22.233690 6665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e: Bootstrap starting.
I20260203 21:07:22.234534 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 6597
I20260203 21:07:22.234632 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:07:22.234843 410 external_mini_cluster.cc:1468] Setting key 2a1f6df6196e75a78a000bc60d5c9a3e
I20260203 21:07:22.236044 6665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:22.236856 6665 log.cc:826] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:22.238616 6665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e: No bootstrap required, opened a new log
I20260203 21:07:22.241120 6665 raft_consensus.cc:359] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d568b8599c4dc490c48b88c56f295e" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 36633 } }
I20260203 21:07:22.241299 6665 raft_consensus.cc:385] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:22.241361 6665 raft_consensus.cc:740] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d568b8599c4dc490c48b88c56f295e, State: Initialized, Role: FOLLOWER
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:07:22.241772 6665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d568b8599c4dc490c48b88c56f295e" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 36633 } }
I20260203 21:07:22.241907 6665 raft_consensus.cc:399] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:07:22.241967 6665 raft_consensus.cc:493] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:07:22.242064 6665 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:22.243034 6665 raft_consensus.cc:515] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d568b8599c4dc490c48b88c56f295e" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 36633 } }
I20260203 21:07:22.243403 6665 leader_election.cc:304] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37d568b8599c4dc490c48b88c56f295e; no voters:
I20260203 21:07:22.243723 6665 leader_election.cc:290] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:07:22.244089 6670 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:22.244266 6670 raft_consensus.cc:697] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [term 1 LEADER]: Becoming Leader. State: Replica: 37d568b8599c4dc490c48b88c56f295e, State: Running, Role: LEADER
I20260203 21:07:22.244560 6670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d568b8599c4dc490c48b88c56f295e" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 36633 } }
I20260203 21:07:22.244860 6665 sys_catalog.cc:565] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:07:22.246006 6671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37d568b8599c4dc490c48b88c56f295e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d568b8599c4dc490c48b88c56f295e" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 36633 } } }
I20260203 21:07:22.246116 6671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:22.246560 6672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37d568b8599c4dc490c48b88c56f295e. Latest consensus state: current_term: 1 leader_uuid: "37d568b8599c4dc490c48b88c56f295e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d568b8599c4dc490c48b88c56f295e" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 36633 } } }
I20260203 21:07:22.247288 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:22.235782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.0.102.190:36633)
0203 21:07:22.236232 (+ 450us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:22.236245 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:22.236287 (+ 42us) server_negotiation.cc:408] Connection header received
0203 21:07:22.236942 (+ 655us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:22.236967 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:22.237313 (+ 346us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:22.237593 (+ 280us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:22.238432 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.239169 (+ 737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:22.239788 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.240038 (+ 250us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:22.242059 (+ 2021us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:22.242083 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:22.242096 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:22.242140 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:22.244095 (+ 1955us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:22.244595 (+ 500us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:22.244601 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:22.244606 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:22.244736 (+ 130us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:22.245014 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:22.245018 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:22.245019 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:22.245412 (+ 393us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:22.245620 (+ 208us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:22.246223 (+ 603us) server_negotiation.cc:300] Negotiation successful
0203 21:07:22.246467 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":147,"threads_started":1}
I20260203 21:07:22.246687 6672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e [sys.catalog]: This master's current role is: LEADER
W20260203 21:07:22.249922 6685 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260203 21:07:22.249987 6685 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260203 21:07:22.250061 6679 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:07:22.250761 6679 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:07:22.255846 6679 catalog_manager.cc:1357] Generated new cluster ID: f2ff3b77622648658286905996c6f619
I20260203 21:07:22.255914 6679 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:07:22.270604 6679 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:07:22.271472 6679 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:07:22.278165 6679 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 37d568b8599c4dc490c48b88c56f295e: Generated new TSK 0
I20260203 21:07:22.278937 6679 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:07:22.327068 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:36633
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:22.429997 6693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:22.430204 6693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:22.430259 6693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:22.434695 6693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:22.434767 6693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:22.434870 6693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:07:22.440057 6693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:22.441125 6693 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:22.441910 6693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:22.448040 6701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:22.448057 6699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:22.448259 6698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:22.448392 6693 server_base.cc:1060] running on GCE node
I20260203 21:07:22.448750 6693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:22.449251 6693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:22.450393 6693 hybrid_clock.cc:648] HybridClock initialized: now 1770152842450373 us; error 35 us; skew 500 ppm
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:22.452867 6693 init.cc:377] Logged in from keytab as kudu/127.0.102.129@KRBTEST.COM (short username kudu)
I20260203 21:07:22.453871 6693 webserver.cc:492] Webserver started at http://127.0.102.129:41059/ using document root <none> and password file <none>
I20260203 21:07:22.454396 6693 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:22.454443 6693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:22.454613 6693 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:22.456259 6693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "d41180f4df26455b9c7912d4f4b8e85e"
format_stamp: "Formatted at 2026-02-03 21:07:22 on dist-test-slave-j889"
server_key: "32dc51a2f2ec4a606cb40e7db8b23ae4"
server_key_iv: "95f79c691f4512aedef704e0ecf5b5e5"
server_key_version: "encryptionkey@0"
I20260203 21:07:22.456722 6693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "d41180f4df26455b9c7912d4f4b8e85e"
format_stamp: "Formatted at 2026-02-03 21:07:22 on dist-test-slave-j889"
server_key: "32dc51a2f2ec4a606cb40e7db8b23ae4"
server_key_iv: "95f79c691f4512aedef704e0ecf5b5e5"
server_key_version: "encryptionkey@0"
I20260203 21:07:22.460137 6693 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260203 21:07:22.462282 6708 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:22.463253 6693 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260203 21:07:22.463357 6693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "d41180f4df26455b9c7912d4f4b8e85e"
format_stamp: "Formatted at 2026-02-03 21:07:22 on dist-test-slave-j889"
server_key: "32dc51a2f2ec4a606cb40e7db8b23ae4"
server_key_iv: "95f79c691f4512aedef704e0ecf5b5e5"
server_key_version: "encryptionkey@0"
I20260203 21:07:22.463441 6693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:22.482988 6693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:22.485492 6693 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:22.485646 6693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:22.486133 6693 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:22.486912 6693 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:22.486955 6693 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:22.487003 6693 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:22.487030 6693 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:22.495633 6693 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:36079
I20260203 21:07:22.495656 6821 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:36079 every 8 connection(s)
I20260203 21:07:22.496536 6693 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:07:22.502261 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 6693
I20260203 21:07:22.502363 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:07:22.502575 410 external_mini_cluster.cc:1468] Setting key 18f67b88d8c6604a469e2457929810ce
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:07:22.508306 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:22.498231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:60883 (local address 127.0.102.190:36633)
0203 21:07:22.498372 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:22.498376 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:22.498993 (+ 617us) server_negotiation.cc:408] Connection header received
0203 21:07:22.499802 (+ 809us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:22.499805 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:22.499861 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:22.499933 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:22.501559 (+ 1626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.502067 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:22.502753 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.502945 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:22.505299 (+ 2354us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:22.505318 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:22.505321 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:22.505344 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:22.506560 (+ 1216us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:22.506994 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:22.506997 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:22.506999 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:22.507042 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:22.507322 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:22.507331 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:22.507333 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:22.507471 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:22.507552 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:22.508019 (+ 467us) server_negotiation.cc:300] Negotiation successful
0203 21:07:22.508137 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260203 21:07:22.508931 6824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:22.498460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:36633 (local address 127.0.102.129:60883)
0203 21:07:22.498867 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:22.498901 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:22.499620 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:22.500106 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:22.500114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:22.500489 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:22.501417 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:22.501431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.502176 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:22.502179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:22.502649 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:22.502658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.502865 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:22.503455 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:22.503480 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:22.505161 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:22.506668 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:22.506673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:22.506685 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:22.506906 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:22.507154 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:22.507156 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:22.507157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:22.507240 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:22.507561 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:22.507568 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:22.507796 (+ 228us) client_negotiation.cc:769] Sending connection context
0203 21:07:22.507992 (+ 196us) client_negotiation.cc:240] Negotiation successful
0203 21:07:22.508249 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.0.102.130@KRBTEST.COM; defaulting to no policy
I20260203 21:07:22.510089 6822 heartbeater.cc:344] Connected to a master server at 127.0.102.190:36633
I20260203 21:07:22.510324 6822 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:22.510819 6822 heartbeater.cc:507] Master 127.0.102.190:36633 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.130@KRBTEST.COM" created.
I20260203 21:07:22.512286 6625 ts_manager.cc:194] Registered new tserver with Master: d41180f4df26455b9c7912d4f4b8e85e (127.0.102.129:36079)
I20260203 21:07:22.513448 6625 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:60883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:07:22.541673 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:36633
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:22.641398 6829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:22.641594 6829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:22.641666 6829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:22.644776 6829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:22.644829 6829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:22.644920 6829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:07:22.649052 6829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:22.650003 6829 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:22.650799 6829 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:22.657063 6835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:22.657047 6834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:22.657121 6837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:22.657289 6829 server_base.cc:1060] running on GCE node
I20260203 21:07:22.657634 6829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:22.658174 6829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:22.659338 6829 hybrid_clock.cc:648] HybridClock initialized: now 1770152842659326 us; error 24 us; skew 500 ppm
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:22.661834 6829 init.cc:377] Logged in from keytab as kudu/127.0.102.130@KRBTEST.COM (short username kudu)
I20260203 21:07:22.662760 6829 webserver.cc:492] Webserver started at http://127.0.102.130:36441/ using document root <none> and password file <none>
I20260203 21:07:22.663246 6829 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:22.663290 6829 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:22.663431 6829 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:22.665052 6829 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "38422be6abbb4604aa1f5b2aef2d4cba"
format_stamp: "Formatted at 2026-02-03 21:07:22 on dist-test-slave-j889"
server_key: "063217e7c192eba008338e3549c473da"
server_key_iv: "5ce54c19a9050f9571a7df841cedbad4"
server_key_version: "encryptionkey@0"
I20260203 21:07:22.665455 6829 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "38422be6abbb4604aa1f5b2aef2d4cba"
format_stamp: "Formatted at 2026-02-03 21:07:22 on dist-test-slave-j889"
server_key: "063217e7c192eba008338e3549c473da"
server_key_iv: "5ce54c19a9050f9571a7df841cedbad4"
server_key_version: "encryptionkey@0"
I20260203 21:07:22.668610 6829 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260203 21:07:22.670574 6844 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:22.671487 6829 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260203 21:07:22.671573 6829 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "38422be6abbb4604aa1f5b2aef2d4cba"
format_stamp: "Formatted at 2026-02-03 21:07:22 on dist-test-slave-j889"
server_key: "063217e7c192eba008338e3549c473da"
server_key_iv: "5ce54c19a9050f9571a7df841cedbad4"
server_key_version: "encryptionkey@0"
I20260203 21:07:22.671643 6829 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:22.699775 6829 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:22.702416 6829 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:22.702564 6829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:22.703094 6829 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:22.703907 6829 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:22.703953 6829 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:22.704010 6829 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:22.704030 6829 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:22.712214 6829 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:40767
I20260203 21:07:22.712255 6957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:40767 every 8 connection(s)
I20260203 21:07:22.713135 6829 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:07:22.716681 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 6829
I20260203 21:07:22.716790 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:07:22.717046 410 external_mini_cluster.cc:1468] Setting key 2c183dcdebb8c18a2219a41f63ee59f0
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:07:22.724761 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:22.714788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:49565 (local address 127.0.102.190:36633)
0203 21:07:22.714911 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:22.714916 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:22.715583 (+ 667us) server_negotiation.cc:408] Connection header received
0203 21:07:22.716361 (+ 778us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:22.716364 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:22.716417 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:22.716509 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:22.718316 (+ 1807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.718796 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:22.719435 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.719620 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:22.721674 (+ 2054us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:22.721689 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:22.721692 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:22.721715 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:22.722943 (+ 1228us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:22.723473 (+ 530us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:22.723475 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:22.723477 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:22.723510 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:22.723782 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:22.723783 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:22.723785 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:22.723905 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:22.723969 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:22.724529 (+ 560us) server_negotiation.cc:300] Negotiation successful
0203 21:07:22.724614 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.0.102.131@KRBTEST.COM; defaulting to no policy
I20260203 21:07:22.725582 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:22.715059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:36633 (local address 127.0.102.130:49565)
0203 21:07:22.715449 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:22.715483 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:22.716177 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:22.716661 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:22.716672 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:22.717084 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:22.718162 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:22.718180 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.718912 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:22.718914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:22.719327 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:22.719335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.719530 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:22.720068 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:22.720091 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:22.721539 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:22.723059 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:22.723065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:22.723076 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:22.723387 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:22.723583 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:22.723586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:22.723588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:22.723704 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:22.724000 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:22.724006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:22.724290 (+ 284us) client_negotiation.cc:769] Sending connection context
0203 21:07:22.724523 (+ 233us) client_negotiation.cc:240] Negotiation successful
0203 21:07:22.724817 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":105,"threads_started":1}
I20260203 21:07:22.726505 6958 heartbeater.cc:344] Connected to a master server at 127.0.102.190:36633
I20260203 21:07:22.726722 6958 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:22.727171 6958 heartbeater.cc:507] Master 127.0.102.190:36633 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.131@KRBTEST.COM" created.
I20260203 21:07:22.728046 6625 ts_manager.cc:194] Registered new tserver with Master: 38422be6abbb4604aa1f5b2aef2d4cba (127.0.102.130:40767)
I20260203 21:07:22.728559 6625 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.130@KRBTEST.COM'} at 127.0.102.130:49565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:07:22.757421 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:36633
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:42793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:22.859912 6965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:22.860112 6965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:22.860172 6965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:22.863385 6965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:22.863436 6965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:22.863529 6965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:07:22.867590 6965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:42793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:22.868566 6965 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:22.869402 6965 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:22.875460 6973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:22.875453 6970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:22.875662 6965 server_base.cc:1060] running on GCE node
W20260203 21:07:22.875453 6971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:22.876127 6965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:22.876573 6965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:22.877710 6965 hybrid_clock.cc:648] HybridClock initialized: now 1770152842877691 us; error 32 us; skew 500 ppm
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:22.880085 6965 init.cc:377] Logged in from keytab as kudu/127.0.102.131@KRBTEST.COM (short username kudu)
I20260203 21:07:22.881031 6965 webserver.cc:492] Webserver started at http://127.0.102.131:44153/ using document root <none> and password file <none>
I20260203 21:07:22.881515 6965 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:22.881563 6965 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:22.881722 6965 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:22.883258 6965 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "fbe568cda4214b8cb281617e4744a985"
format_stamp: "Formatted at 2026-02-03 21:07:22 on dist-test-slave-j889"
server_key: "94f7ec25aa28dd86d011c0488e7eb957"
server_key_iv: "6cd03d70e5d3b9c8e3454d8a4ca43ac8"
server_key_version: "encryptionkey@0"
I20260203 21:07:22.883689 6965 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "fbe568cda4214b8cb281617e4744a985"
format_stamp: "Formatted at 2026-02-03 21:07:22 on dist-test-slave-j889"
server_key: "94f7ec25aa28dd86d011c0488e7eb957"
server_key_iv: "6cd03d70e5d3b9c8e3454d8a4ca43ac8"
server_key_version: "encryptionkey@0"
I20260203 21:07:22.886720 6965 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260203 21:07:22.888576 6980 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:22.889500 6965 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:22.889595 6965 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "fbe568cda4214b8cb281617e4744a985"
format_stamp: "Formatted at 2026-02-03 21:07:22 on dist-test-slave-j889"
server_key: "94f7ec25aa28dd86d011c0488e7eb957"
server_key_iv: "6cd03d70e5d3b9c8e3454d8a4ca43ac8"
server_key_version: "encryptionkey@0"
I20260203 21:07:22.889669 6965 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:22.909265 6965 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:22.911532 6965 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:22.911690 6965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:22.912196 6965 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:22.913012 6965 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:22.913064 6965 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:22.913115 6965 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:22.913142 6965 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:22.921226 6965 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:34945
I20260203 21:07:22.921257 7093 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:34945 every 8 connection(s)
I20260203 21:07:22.922068 6965 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:07:22.931835 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 6965
I20260203 21:07:22.931937 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:07:22.932161 410 external_mini_cluster.cc:1468] Setting key beddc60f8002f7acfa3bea62a454937d
I20260203 21:07:22.932955 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:22.923585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:34407 (local address 127.0.102.190:36633)
0203 21:07:22.923719 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:22.923723 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:22.924324 (+ 601us) server_negotiation.cc:408] Connection header received
0203 21:07:22.925141 (+ 817us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:22.925145 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:22.925200 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:22.925276 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:22.926794 (+ 1518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.927287 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:22.927911 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.928088 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:22.929997 (+ 1909us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:22.930014 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:22.930016 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:22.930039 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:22.931112 (+ 1073us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:22.931543 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:22.931547 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:22.931548 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:22.931590 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:22.931849 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:22.931852 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:22.931854 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:22.932035 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:22.932126 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:22.932702 (+ 576us) server_negotiation.cc:300] Negotiation successful
0203 21:07:22.932828 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260203 21:07:22.933665 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:22.923807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:36633 (local address 127.0.102.131:34407)
0203 21:07:22.924194 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:22.924226 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:22.924956 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:22.925432 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:22.925439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:22.925780 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:22.926652 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:22.926665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.927410 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:22.927413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:22.927803 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:22.927809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.927997 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:22.928527 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:22.928549 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:22.929862 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:22.931220 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:22.931226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:22.931237 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:22.931453 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:22.931684 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:22.931687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:22.931688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:22.931768 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:22.932128 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:22.932135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:22.932390 (+ 255us) client_negotiation.cc:769] Sending connection context
0203 21:07:22.932605 (+ 215us) client_negotiation.cc:240] Negotiation successful
0203 21:07:22.932889 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":97,"threads_started":1}
I20260203 21:07:22.934613 7094 heartbeater.cc:344] Connected to a master server at 127.0.102.190:36633
I20260203 21:07:22.934820 7094 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:22.935295 7094 heartbeater.cc:507] Master 127.0.102.190:36633 requested a full tablet report, sending...
I20260203 21:07:22.936122 6625 ts_manager.cc:194] Registered new tserver with Master: fbe568cda4214b8cb281617e4744a985 (127.0.102.131:34945)
I20260203 21:07:22.936674 6625 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.131@KRBTEST.COM'} at 127.0.102.131:34407
I20260203 21:07:22.945569 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/test-admin.keytab.
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:22.958470 410 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260203 21:07:22.960700 7101 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:22.959594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44978 (local address 127.0.102.129:36079)
0203 21:07:22.959876 (+ 282us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:22.959881 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:22.959902 (+ 21us) server_negotiation.cc:408] Connection header received
0203 21:07:22.959948 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:22.959952 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:22.960107 (+ 155us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:22.960214 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:22.960530 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44978: BlockingRecv error: recv got EOF from 127.0.0.1:44978 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":105,"threads_started":1}
W20260203 21:07:22.960896 410 init.cc:244] Error while opening credential cache '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/krb5cc)
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:22.961774 410 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:22.962584 410 init.cc:295] Successfully reacquired a new kerberos TGT
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.129@KRBTEST.COM
I20260203 21:07:22.970516 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:22.963328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.0.102.129:36079)
0203 21:07:22.963443 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:22.963447 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:22.963463 (+ 16us) server_negotiation.cc:408] Connection header received
0203 21:07:22.963588 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:22.963591 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:22.963629 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:22.963688 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:22.964352 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.965120 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:22.965760 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.965979 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:22.967610 (+ 1631us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:22.967628 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:22.967638 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:22.967660 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:22.969202 (+ 1542us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:22.969519 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:22.969523 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:22.969525 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:22.969572 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:22.969789 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:22.969792 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:22.969793 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:22.969958 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:22.970129 (+ 171us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:22.970293 (+ 164us) server_negotiation.cc:300] Negotiation successful
0203 21:07:22.970406 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260203 21:07:22.973832 7101 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:22.973086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.0.102.129:36079)
0203 21:07:22.973204 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:22.973207 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:22.973221 (+ 14us) server_negotiation.cc:408] Connection header received
0203 21:07:22.973334 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:22.973337 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:22.973374 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:22.973436 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:22.973742 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44984: BlockingRecv error: recv got EOF from 127.0.0.1:44984 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260203 21:07:22.974069 410 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/krb5cc' of type FILE
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:22.975096 410 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:22.975915 410 init.cc:295] Successfully reacquired a new kerberos TGT
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.129@KRBTEST.COM
I20260203 21:07:22.983204 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:22.976739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45000 (local address 127.0.102.129:36079)
0203 21:07:22.976844 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:22.976848 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:22.976957 (+ 109us) server_negotiation.cc:408] Connection header received
0203 21:07:22.977091 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:22.977095 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:22.977135 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:22.977199 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:22.977741 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.978238 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:22.978804 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.978994 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:22.980666 (+ 1672us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:22.980683 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:22.980686 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:22.980707 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:22.981912 (+ 1205us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:22.982290 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:22.982294 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:22.982295 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:22.982336 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:22.982528 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:22.982532 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:22.982534 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:22.982656 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:22.982764 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:22.982963 (+ 199us) server_negotiation.cc:300] Negotiation successful
0203 21:07:22.983057 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260203 21:07:22.985500 7101 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:22.984618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45016 (local address 127.0.102.129:36079)
0203 21:07:22.984751 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:22.984754 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:22.984831 (+ 77us) server_negotiation.cc:408] Connection header received
0203 21:07:22.984887 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:22.984889 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:22.984924 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:22.984994 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:22.985416 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45016: BlockingRecv error: recv got EOF from 127.0.0.1:45016 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260203 21:07:22.985615 410 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1770152788143875-410-0/krb5kdc/krb5cc' of type FILE
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:22.986510 410 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:22.987246 410 init.cc:295] Successfully reacquired a new kerberos TGT
Feb 03 21:07:22 dist-test-slave-j889 krb5kdc[6581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.129@KRBTEST.COM
I20260203 21:07:22.993868 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:22.987929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.0.102.129:36079)
0203 21:07:22.988054 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:22.988058 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:22.988100 (+ 42us) server_negotiation.cc:408] Connection header received
0203 21:07:22.988212 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:22.988215 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:22.988249 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:22.988310 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:22.988856 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.989288 (+ 432us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:22.989831 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:22.990003 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:22.991622 (+ 1619us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:22.991635 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:22.991637 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:22.991657 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:22.992820 (+ 1163us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:22.993129 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:22.993131 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:22.993132 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:22.993174 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:22.993343 (+ 169us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:22.993345 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:22.993346 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:22.993448 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:22.993528 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:22.993696 (+ 168us) server_negotiation.cc:300] Negotiation successful
0203 21:07:22.993771 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260203 21:07:22.994733 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 6693
I20260203 21:07:22.999640 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 6829
I20260203 21:07:23.004763 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 6965
I20260203 21:07:23.009188 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 6597
2026-02-03T21:07:23Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2914 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:07:23 dist-test-slave-j889 krb5kdc[7114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:07:23 dist-test-slave-j889 krb5kdc[7114](info): set up 2 sockets
Feb 03 21:07:23 dist-test-slave-j889 krb5kdc[7114](info): commencing operation
krb5kdc: starting...
W20260203 21:07:24.833001 410 mini_kdc.cc:121] Time spent starting KDC: real 1.804s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:07:24 dist-test-slave-j889 krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:07:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:07:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
I20260203 21:07:24.961710 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:42297
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:42297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:25.062623 7130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:25.062851 7130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:25.062927 7130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:25.066226 7130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:07:25.066285 7130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:25.066315 7130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:25.066356 7130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:07:25.066390 7130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:07:25.070560 7130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:42297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:42297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:25.071553 7130 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:25.072352 7130 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:25.077638 7135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:25.077630 7138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:25.077630 7136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:25.077908 7130 server_base.cc:1060] running on GCE node
I20260203 21:07:25.078347 7130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:25.079133 7130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:25.080271 7130 hybrid_clock.cc:648] HybridClock initialized: now 1770152845080259 us; error 27 us; skew 500 ppm
Feb 03 21:07:25 dist-test-slave-j889 krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:25.082623 7130 init.cc:377] Logged in from keytab as oryx/127.0.102.190@KRBTEST.COM (short username oryx)
I20260203 21:07:25.083572 7130 webserver.cc:492] Webserver started at http://127.0.102.190:45195/ using document root <none> and password file <none>
I20260203 21:07:25.084084 7130 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:25.084131 7130 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:25.084303 7130 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:25.085887 7130 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "61f5416a4138441e8bdfa4f27aeb963d"
format_stamp: "Formatted at 2026-02-03 21:07:25 on dist-test-slave-j889"
server_key: "f722d29c6b55aca8bab50c0218ea9252"
server_key_iv: "d51d407279082c20fff43d9f9df1d897"
server_key_version: "encryptionkey@0"
I20260203 21:07:25.086308 7130 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "61f5416a4138441e8bdfa4f27aeb963d"
format_stamp: "Formatted at 2026-02-03 21:07:25 on dist-test-slave-j889"
server_key: "f722d29c6b55aca8bab50c0218ea9252"
server_key_iv: "d51d407279082c20fff43d9f9df1d897"
server_key_version: "encryptionkey@0"
I20260203 21:07:25.089325 7130 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260203 21:07:25.091276 7145 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:25.092172 7130 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260203 21:07:25.092268 7130 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "61f5416a4138441e8bdfa4f27aeb963d"
format_stamp: "Formatted at 2026-02-03 21:07:25 on dist-test-slave-j889"
server_key: "f722d29c6b55aca8bab50c0218ea9252"
server_key_iv: "d51d407279082c20fff43d9f9df1d897"
server_key_version: "encryptionkey@0"
I20260203 21:07:25.092343 7130 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:25.106688 7130 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:25.113283 7130 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:25.113471 7130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:25.120311 7130 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:42297
I20260203 21:07:25.120309 7197 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:42297 every 8 connection(s)
I20260203 21:07:25.121374 7130 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:07:25.123999 7198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:25.126631 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 7130
I20260203 21:07:25.126745 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:07:25.126962 410 external_mini_cluster.cc:1468] Setting key dd08f8b6417f8682909f262832c0b878
I20260203 21:07:25.129473 7198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d: Bootstrap starting.
I20260203 21:07:25.131410 7198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:25.132023 7198 log.cc:826] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d: Log is configured to *not* fsync() on all Append() calls
Feb 03 21:07:25 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM
I20260203 21:07:25.133845 7198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d: No bootstrap required, opened a new log
I20260203 21:07:25.136238 7198 raft_consensus.cc:359] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } }
I20260203 21:07:25.136412 7198 raft_consensus.cc:385] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:25.136462 7198 raft_consensus.cc:740] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f5416a4138441e8bdfa4f27aeb963d, State: Initialized, Role: FOLLOWER
I20260203 21:07:25.136879 7198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } }
I20260203 21:07:25.136993 7198 raft_consensus.cc:399] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:07:25.137070 7198 raft_consensus.cc:493] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:07:25.137176 7198 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:25.137979 7198 raft_consensus.cc:515] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } }
I20260203 21:07:25.138160 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:25.128105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.0.102.190:42297)
0203 21:07:25.128478 (+ 373us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:25.128487 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:25.128518 (+ 31us) server_negotiation.cc:408] Connection header received
0203 21:07:25.129059 (+ 541us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:25.129075 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:25.129354 (+ 279us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:25.129637 (+ 283us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:25.130499 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:25.131223 (+ 724us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:25.131873 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:25.132133 (+ 260us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:25.134289 (+ 2156us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:25.134306 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:25.134316 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:25.134348 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:25.135850 (+ 1502us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:25.136201 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:25.136205 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:25.136210 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:25.136278 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:25.136510 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:25.136514 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:25.136516 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:25.136910 (+ 394us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:25.137106 (+ 196us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:25.137387 (+ 281us) server_negotiation.cc:300] Negotiation successful
0203 21:07:25.137602 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
I20260203 21:07:25.138324 7198 leader_election.cc:304] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61f5416a4138441e8bdfa4f27aeb963d; no voters:
I20260203 21:07:25.138589 7198 leader_election.cc:290] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:07:25.138942 7203 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:25.139114 7203 raft_consensus.cc:697] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [term 1 LEADER]: Becoming Leader. State: Replica: 61f5416a4138441e8bdfa4f27aeb963d, State: Running, Role: LEADER
I20260203 21:07:25.139451 7203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } }
I20260203 21:07:25.139729 7198 sys_catalog.cc:565] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:07:25.140740 7205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61f5416a4138441e8bdfa4f27aeb963d. Latest consensus state: current_term: 1 leader_uuid: "61f5416a4138441e8bdfa4f27aeb963d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } }
I20260203 21:07:25.140841 7205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:25.141036 7204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61f5416a4138441e8bdfa4f27aeb963d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } }
I20260203 21:07:25.141124 7204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:25.141314 7212 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:07:25.143950 7212 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:07:25.148955 7212 catalog_manager.cc:1357] Generated new cluster ID: 236be3fb335142d6a4092ad9c8733b3a
I20260203 21:07:25.149009 7212 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:07:25.166563 7212 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:07:25.167735 7212 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:07:25.174290 7212 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d: Generated new TSK 0
I20260203 21:07:25.174917 7212 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
I20260203 21:07:25.224046 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:42297
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:25.329461 7226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:25.329687 7226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:25.329744 7226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:25.332973 7226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:25.333029 7226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:25.333108 7226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:07:25.337149 7226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:25.338043 7226 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:25.338881 7226 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:25.345363 7231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:25.345398 7232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:25.345535 7226 server_base.cc:1060] running on GCE node
W20260203 21:07:25.345400 7234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:25.346045 7226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:25.346572 7226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:25.347730 7226 hybrid_clock.cc:648] HybridClock initialized: now 1770152845347717 us; error 28 us; skew 500 ppm
Feb 03 21:07:25 dist-test-slave-j889 krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:25.350173 7226 init.cc:377] Logged in from keytab as oryx/127.0.102.129@KRBTEST.COM (short username oryx)
I20260203 21:07:25.351158 7226 webserver.cc:492] Webserver started at http://127.0.102.129:34797/ using document root <none> and password file <none>
I20260203 21:07:25.351665 7226 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:25.351711 7226 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:25.351866 7226 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:25.353559 7226 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "f1f9e3fc5a4e44edaca392a6e083f178"
format_stamp: "Formatted at 2026-02-03 21:07:25 on dist-test-slave-j889"
server_key: "1583637660e9c74e2a3d474b34e41cc1"
server_key_iv: "5f684fc980a1aee2b88cf587d36ac460"
server_key_version: "encryptionkey@0"
I20260203 21:07:25.353993 7226 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "f1f9e3fc5a4e44edaca392a6e083f178"
format_stamp: "Formatted at 2026-02-03 21:07:25 on dist-test-slave-j889"
server_key: "1583637660e9c74e2a3d474b34e41cc1"
server_key_iv: "5f684fc980a1aee2b88cf587d36ac460"
server_key_version: "encryptionkey@0"
I20260203 21:07:25.357357 7226 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260203 21:07:25.359480 7241 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:25.360477 7226 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:25.360570 7226 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "f1f9e3fc5a4e44edaca392a6e083f178"
format_stamp: "Formatted at 2026-02-03 21:07:25 on dist-test-slave-j889"
server_key: "1583637660e9c74e2a3d474b34e41cc1"
server_key_iv: "5f684fc980a1aee2b88cf587d36ac460"
server_key_version: "encryptionkey@0"
I20260203 21:07:25.360694 7226 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:25.391103 7226 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:25.393926 7226 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:25.394073 7226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:25.394547 7226 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:25.395367 7226 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:25.395414 7226 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:25.395452 7226 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:25.395465 7226 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:25.404068 7226 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:36543
I20260203 21:07:25.404091 7354 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:36543 every 8 connection(s)
I20260203 21:07:25.404992 7226 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:07:25.409477 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 7226
I20260203 21:07:25.409593 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:07:25.409803 410 external_mini_cluster.cc:1468] Setting key 3fa9495c4ac3ed6400176d611ece36eb
Feb 03 21:07:25 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM
I20260203 21:07:25.416988 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:25.406612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:57309 (local address 127.0.102.190:42297)
0203 21:07:25.406755 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:25.406760 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:25.407422 (+ 662us) server_negotiation.cc:408] Connection header received
0203 21:07:25.408186 (+ 764us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:25.408189 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:25.408235 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:25.408304 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:25.409992 (+ 1688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:25.410580 (+ 588us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:25.411247 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:25.411457 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:25.413613 (+ 2156us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:25.413630 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:25.413632 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:25.413656 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:25.414944 (+ 1288us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:25.415500 (+ 556us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:25.415503 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:25.415504 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:25.415547 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:25.415886 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:25.415889 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:25.415890 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:25.416051 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:25.416138 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:25.416754 (+ 616us) server_negotiation.cc:300] Negotiation successful
0203 21:07:25.416869 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260203 21:07:25.417853 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:25.406880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.129:57309)
0203 21:07:25.407289 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:25.407324 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:25.408020 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:25.408445 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:25.408453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:25.408878 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:25.409824 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:25.409840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:25.410710 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:25.410713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:25.411143 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:25.411151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:25.411349 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:25.411885 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:25.411910 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:25.413453 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:25.415062 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:25.415068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:25.415080 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:25.415401 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:25.415636 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:25.415640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:25.415642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:25.415773 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:25.416155 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:25.416162 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:25.416465 (+ 303us) client_negotiation.cc:769] Sending connection context
0203 21:07:25.416733 (+ 268us) client_negotiation.cc:240] Negotiation successful
0203 21:07:25.417004 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for oryx/127.0.102.130@KRBTEST.COM; defaulting to no policy
I20260203 21:07:25.418991 7355 heartbeater.cc:344] Connected to a master server at 127.0.102.190:42297
I20260203 21:07:25.419205 7355 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:25.419744 7355 heartbeater.cc:507] Master 127.0.102.190:42297 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.102.130@KRBTEST.COM" created.
I20260203 21:07:25.421295 7162 ts_manager.cc:194] Registered new tserver with Master: f1f9e3fc5a4e44edaca392a6e083f178 (127.0.102.129:36543)
I20260203 21:07:25.422710 7162 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:57309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
I20260203 21:07:25.455709 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:42297
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:25.562510 7362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:25.562747 7362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:25.562806 7362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:25.566155 7362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:25.566222 7362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:25.566313 7362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:07:25.570552 7362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:25.571595 7362 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:25.572438 7362 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:25.578958 7370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:25.578958 7368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:25.579052 7362 server_base.cc:1060] running on GCE node
W20260203 21:07:25.579185 7367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:25.579522 7362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:25.580060 7362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:25.581210 7362 hybrid_clock.cc:648] HybridClock initialized: now 1770152845581186 us; error 42 us; skew 500 ppm
Feb 03 21:07:25 dist-test-slave-j889 krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:25.583902 7362 init.cc:377] Logged in from keytab as oryx/127.0.102.130@KRBTEST.COM (short username oryx)
I20260203 21:07:25.584975 7362 webserver.cc:492] Webserver started at http://127.0.102.130:37805/ using document root <none> and password file <none>
I20260203 21:07:25.585497 7362 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:25.585544 7362 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:25.585736 7362 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:25.587396 7362 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f"
format_stamp: "Formatted at 2026-02-03 21:07:25 on dist-test-slave-j889"
server_key: "fd0f3ac7a6f709107d786045f5b14964"
server_key_iv: "8999dc38d823f8326df9ef149e07494d"
server_key_version: "encryptionkey@0"
I20260203 21:07:25.587847 7362 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f"
format_stamp: "Formatted at 2026-02-03 21:07:25 on dist-test-slave-j889"
server_key: "fd0f3ac7a6f709107d786045f5b14964"
server_key_iv: "8999dc38d823f8326df9ef149e07494d"
server_key_version: "encryptionkey@0"
I20260203 21:07:25.591153 7362 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260203 21:07:25.593228 7377 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:25.594154 7362 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:25.594251 7362 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f"
format_stamp: "Formatted at 2026-02-03 21:07:25 on dist-test-slave-j889"
server_key: "fd0f3ac7a6f709107d786045f5b14964"
server_key_iv: "8999dc38d823f8326df9ef149e07494d"
server_key_version: "encryptionkey@0"
I20260203 21:07:25.594328 7362 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:25.614430 7362 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:25.616918 7362 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:25.617074 7362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:25.617559 7362 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:25.618361 7362 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:25.618413 7362 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:25.618463 7362 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:25.618490 7362 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:25.627100 7362 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:45287
I20260203 21:07:25.627133 7490 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:45287 every 8 connection(s)
I20260203 21:07:25.628014 7362 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:07:25.631209 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 7362
I20260203 21:07:25.631333 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:07:25.631559 410 external_mini_cluster.cc:1468] Setting key d72510ed8cdd233a57524a6fdf9b634e
Feb 03 21:07:25 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.102.131@KRBTEST.COM; defaulting to no policy
I20260203 21:07:25.639228 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:25.629625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:49721 (local address 127.0.102.190:42297)
0203 21:07:25.629754 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:25.629757 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:25.630372 (+ 615us) server_negotiation.cc:408] Connection header received
0203 21:07:25.631243 (+ 871us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:25.631246 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:25.631301 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:25.631379 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:25.633042 (+ 1663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:25.633530 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:25.634120 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:25.634290 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:25.636248 (+ 1958us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:25.636262 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:25.636264 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:25.636287 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:25.637567 (+ 1280us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:25.637995 (+ 428us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:25.637998 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:25.637999 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:25.638042 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:25.638300 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:25.638303 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:25.638305 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:25.638448 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:25.638528 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:25.638971 (+ 443us) server_negotiation.cc:300] Negotiation successful
0203 21:07:25.639083 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260203 21:07:25.639806 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:25.629859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.130:49721)
0203 21:07:25.630246 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:25.630280 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:25.631017 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:25.631545 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:25.631553 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:25.631942 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:25.632909 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:25.632927 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:25.633643 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:25.633646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:25.634018 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:25.634025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:25.634213 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:25.634773 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:25.634796 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:25.636117 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:25.637674 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:25.637684 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:25.637694 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:25.637908 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:25.638113 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:25.638116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:25.638118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:25.638206 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:25.638539 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:25.638545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:25.638759 (+ 214us) client_negotiation.cc:769] Sending connection context
0203 21:07:25.638926 (+ 167us) client_negotiation.cc:240] Negotiation successful
0203 21:07:25.639162 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":96,"threads_started":1}
I20260203 21:07:25.640833 7491 heartbeater.cc:344] Connected to a master server at 127.0.102.190:42297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.102.131@KRBTEST.COM" created.
I20260203 21:07:25.641058 7491 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:25.641606 7491 heartbeater.cc:507] Master 127.0.102.190:42297 requested a full tablet report, sending...
I20260203 21:07:25.642546 7162 ts_manager.cc:194] Registered new tserver with Master: 2e5f9dfb6e7a4fc0bb51f68658d4b05f (127.0.102.130:45287)
I20260203 21:07:25.643049 7162 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.102.130@KRBTEST.COM'} at 127.0.102.130:49721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
I20260203 21:07:25.671273 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:42297
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:25.772526 7498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:25.772787 7498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:25.772862 7498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:25.776070 7498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:25.776125 7498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:25.776219 7498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:07:25.780282 7498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:25.781337 7498 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:25.782171 7498 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:25.788617 7503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:25.788619 7504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:25.788619 7506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:25.788813 7498 server_base.cc:1060] running on GCE node
I20260203 21:07:25.789256 7498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:25.789784 7498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:25.790918 7498 hybrid_clock.cc:648] HybridClock initialized: now 1770152845790908 us; error 29 us; skew 500 ppm
Feb 03 21:07:25 dist-test-slave-j889 krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:25.793372 7498 init.cc:377] Logged in from keytab as oryx/127.0.102.131@KRBTEST.COM (short username oryx)
I20260203 21:07:25.794340 7498 webserver.cc:492] Webserver started at http://127.0.102.131:40259/ using document root <none> and password file <none>
I20260203 21:07:25.794847 7498 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:25.794894 7498 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:25.795058 7498 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:25.796691 7498 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "a8644652b1a349dbbb003fe66b88e94f"
format_stamp: "Formatted at 2026-02-03 21:07:25 on dist-test-slave-j889"
server_key: "33bcee75ca232c36e5f843740a47ea11"
server_key_iv: "7a389bdc37e446af688fdccda5970ea1"
server_key_version: "encryptionkey@0"
I20260203 21:07:25.797133 7498 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "a8644652b1a349dbbb003fe66b88e94f"
format_stamp: "Formatted at 2026-02-03 21:07:25 on dist-test-slave-j889"
server_key: "33bcee75ca232c36e5f843740a47ea11"
server_key_iv: "7a389bdc37e446af688fdccda5970ea1"
server_key_version: "encryptionkey@0"
I20260203 21:07:25.800282 7498 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260203 21:07:25.802263 7513 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:25.803153 7498 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260203 21:07:25.803246 7498 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "a8644652b1a349dbbb003fe66b88e94f"
format_stamp: "Formatted at 2026-02-03 21:07:25 on dist-test-slave-j889"
server_key: "33bcee75ca232c36e5f843740a47ea11"
server_key_iv: "7a389bdc37e446af688fdccda5970ea1"
server_key_version: "encryptionkey@0"
I20260203 21:07:25.803323 7498 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:25.824616 7498 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:25.827453 7498 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:25.827627 7498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:25.828146 7498 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:25.829200 7498 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:25.829257 7498 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:25.829316 7498 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:25.829350 7498 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:25.837899 7498 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:34113
I20260203 21:07:25.837929 7626 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:34113 every 8 connection(s)
I20260203 21:07:25.838780 7498 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
I20260203 21:07:25.846454 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 7498
I20260203 21:07:25.846580 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:07:25.846835 410 external_mini_cluster.cc:1468] Setting key 1996c45fe009061ccfd2695e206dc03b
Feb 03 21:07:25 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM
I20260203 21:07:25.850960 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:25.840335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:53639 (local address 127.0.102.190:42297)
0203 21:07:25.840494 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:25.840498 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:25.841156 (+ 658us) server_negotiation.cc:408] Connection header received
0203 21:07:25.842035 (+ 879us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:25.842039 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:25.842095 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:25.842166 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:25.843746 (+ 1580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:25.844359 (+ 613us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:25.845087 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:25.845261 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:25.847805 (+ 2544us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:25.847822 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:25.847824 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:25.847851 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:25.849200 (+ 1349us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:25.849647 (+ 447us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:25.849651 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:25.849652 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:25.849702 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:25.849960 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:25.849963 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:25.849965 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:25.850108 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:25.850191 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:25.850664 (+ 473us) server_negotiation.cc:300] Negotiation successful
0203 21:07:25.850795 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260203 21:07:25.851538 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:25.840568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.131:53639)
0203 21:07:25.841032 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:25.841065 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:25.841850 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:25.842323 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:25.842331 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:25.842694 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:25.843594 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:25.843608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:25.844472 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:25.844475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:25.844981 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:25.844990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:25.845197 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:25.845784 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:25.845818 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:25.847610 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:25.849304 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:25.849310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:25.849320 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:25.849558 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:25.849778 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:25.849781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:25.849783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:25.849879 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:25.850199 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:25.850206 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:25.850438 (+ 232us) client_negotiation.cc:769] Sending connection context
0203 21:07:25.850616 (+ 178us) client_negotiation.cc:240] Negotiation successful
0203 21:07:25.850863 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":96,"threads_started":1}
I20260203 21:07:25.852479 7627 heartbeater.cc:344] Connected to a master server at 127.0.102.190:42297
I20260203 21:07:25.852712 7627 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:25.853160 7627 heartbeater.cc:507] Master 127.0.102.190:42297 requested a full tablet report, sending...
I20260203 21:07:25.854055 7162 ts_manager.cc:194] Registered new tserver with Master: a8644652b1a349dbbb003fe66b88e94f (127.0.102.131:34113)
I20260203 21:07:25.854579 7162 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.102.131@KRBTEST.COM'} at 127.0.102.131:53639
I20260203 21:07:25.860375 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.102.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.102.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.102.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.102.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.102.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.102.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
I20260203 21:07:25.905081 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.189:38693
--webserver_interface=127.0.102.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:42297,127.0.102.189:38693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:26.009274 7634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:26.009509 7634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:26.009578 7634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:26.012815 7634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:07:26.012878 7634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:26.012909 7634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:26.012943 7634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:07:26.012974 7634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:07:26.017114 7634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:42297,127.0.102.189:38693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.189:38693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.102.189
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
W20260203 21:07:26.017309 7634 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.102.190:42297,127.0.102.189:38693'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260203 21:07:26.018160 7634 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:26.018996 7634 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:26.024686 7640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:26.024694 7639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:26.024812 7634 server_base.cc:1060] running on GCE node
W20260203 21:07:26.024695 7642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:26.025557 7634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:26.026491 7634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:26.027649 7634 hybrid_clock.cc:648] HybridClock initialized: now 1770152846027630 us; error 36 us; skew 500 ppm
Feb 03 21:07:26 dist-test-slave-j889 krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:26.030244 7634 init.cc:377] Logged in from keytab as oryx/127.0.102.189@KRBTEST.COM (short username oryx)
I20260203 21:07:26.031356 7634 webserver.cc:492] Webserver started at http://127.0.102.189:34571/ using document root <none> and password file <none>
I20260203 21:07:26.031875 7634 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:26.031919 7634 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:26.032070 7634 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:26.033751 7634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/data/instance:
uuid: "c31601e9963b484bbb7869e10d137467"
format_stamp: "Formatted at 2026-02-03 21:07:26 on dist-test-slave-j889"
server_key: "7d6f6d7c96a584bc38de1754ac71b0b4"
server_key_iv: "f48589ed72693a3a5a22dbcc1f211630"
server_key_version: "encryptionkey@0"
I20260203 21:07:26.034180 7634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/wal/instance:
uuid: "c31601e9963b484bbb7869e10d137467"
format_stamp: "Formatted at 2026-02-03 21:07:26 on dist-test-slave-j889"
server_key: "7d6f6d7c96a584bc38de1754ac71b0b4"
server_key_iv: "f48589ed72693a3a5a22dbcc1f211630"
server_key_version: "encryptionkey@0"
I20260203 21:07:26.037509 7634 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260203 21:07:26.039647 7649 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:26.040753 7634 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:07:26.040856 7634 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/wal
uuid: "c31601e9963b484bbb7869e10d137467"
format_stamp: "Formatted at 2026-02-03 21:07:26 on dist-test-slave-j889"
server_key: "7d6f6d7c96a584bc38de1754ac71b0b4"
server_key_iv: "f48589ed72693a3a5a22dbcc1f211630"
server_key_version: "encryptionkey@0"
I20260203 21:07:26.040943 7634 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:26.052500 7634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:26.055295 7634 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:26.055488 7634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:26.062542 7634 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.189:38693
I20260203 21:07:26.062602 7701 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.189:38693 every 8 connection(s)
I20260203 21:07:26.063577 7634 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/data/info.pb
I20260203 21:07:26.066373 7702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:26.070413 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 7634
I20260203 21:07:26.070279 7702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } has no permanent_uuid. Determining permanent_uuid...
Feb 03 21:07:26 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.189@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM
Feb 03 21:07:26 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.102.189@KRBTEST.COM
I20260203 21:07:26.081204 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.072538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54454 (local address 127.0.102.190:42297)
0203 21:07:26.072694 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.072699 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.072890 (+ 191us) server_negotiation.cc:408] Connection header received
0203 21:07:26.073250 (+ 360us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.073253 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.073309 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.073385 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:26.074297 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.074799 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.075479 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.075665 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.077845 (+ 2180us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:26.077868 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:26.077871 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:26.077902 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:26.079226 (+ 1324us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.079695 (+ 469us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.079698 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.079699 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.079745 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.079986 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.079989 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.079991 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.080201 (+ 210us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:26.080302 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.080918 (+ 616us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.081052 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260203 21:07:26.081895 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.072399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:54454)
0203 21:07:26.072763 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.072804 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.073066 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.073439 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.073444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.073870 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:26.074149 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.074165 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.074891 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.074894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.075347 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.075355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.075531 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.076170 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:26.076195 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:26.077648 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:26.079345 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.079352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.079361 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.079602 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.079823 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.079827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.079829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.079908 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.080311 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:26.080317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:26.080620 (+ 303us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.080872 (+ 252us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.081142 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":98,"threads_started":1}
I20260203 21:07:26.081923 7705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.071845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.0.102.189:38693)
0203 21:07:26.072241 (+ 396us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.072250 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.072281 (+ 31us) server_negotiation.cc:408] Connection header received
0203 21:07:26.072944 (+ 663us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.072970 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.073175 (+ 205us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.073323 (+ 148us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:26.074177 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.074869 (+ 692us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.075512 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.075707 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.078211 (+ 2504us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:26.078231 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:26.078242 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:26.078266 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:26.080023 (+ 1757us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.080474 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.080479 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.080481 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.080539 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.080841 (+ 302us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.080844 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.080847 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.081051 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:26.081231 (+ 180us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.081705 (+ 474us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.081807 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":124,"threads_started":1}
I20260203 21:07:26.083674 7702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } has no permanent_uuid. Determining permanent_uuid...
Feb 03 21:07:26 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.189@KRBTEST.COM for oryx/127.0.102.189@KRBTEST.COM
I20260203 21:07:26.089964 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.084121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.189:38693 (local address 127.0.0.1:53300)
0203 21:07:26.084234 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.084248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.084309 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.084488 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.084490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.084613 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:26.084837 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.084843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.085480 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.085484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.085843 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.085850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.085946 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.086351 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:26.086363 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:26.087491 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:26.088938 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.088941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.088942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.089085 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.089242 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.089244 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.089245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.089280 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.089538 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:26.089541 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:26.089605 (+ 64us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.089686 (+ 81us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.089794 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260203 21:07:26.089968 7705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.084187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.0.102.189:38693)
0203 21:07:26.084284 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.084287 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.084298 (+ 11us) server_negotiation.cc:408] Connection header received
0203 21:07:26.084372 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.084374 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.084408 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.084468 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:26.084931 (+ 463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.085373 (+ 442us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.085946 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.086087 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.087603 (+ 1516us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:26.087618 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:26.087620 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:26.087640 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:26.088849 (+ 1209us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.089153 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.089156 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.089157 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.089188 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.089335 (+ 147us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.089337 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.089338 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.089453 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:26.089536 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.089748 (+ 212us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.089850 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":38}
I20260203 21:07:26.092463 7702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: Bootstrap starting.
I20260203 21:07:26.094705 7702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:26.095340 7702 log.cc:826] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:26.096962 7702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: No bootstrap required, opened a new log
I20260203 21:07:26.099411 7702 raft_consensus.cc:359] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } }
I20260203 21:07:26.099566 7702 raft_consensus.cc:385] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:26.099601 7702 raft_consensus.cc:740] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c31601e9963b484bbb7869e10d137467, State: Initialized, Role: FOLLOWER
I20260203 21:07:26.099979 7702 consensus_queue.cc:260] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } }
I20260203 21:07:26.100373 7708 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } } }
I20260203 21:07:26.100486 7708 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: This master's current role is: FOLLOWER
I20260203 21:07:26.101002 7702 sys_catalog.cc:565] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: configured and running, proceeding with master startup.
W20260203 21:07:26.105440 7719 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260203 21:07:26.105530 7719 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260203 21:07:26.107203 7634 master_runner.cc:428] Detected that this master c31601e9963b484bbb7869e10d137467 is joining an existing cluster
I20260203 21:07:26.107272 7634 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.102.189:38693
I20260203 21:07:26.116885 7705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.109550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53306 (local address 127.0.102.189:38693)
0203 21:07:26.109664 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.109668 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.110081 (+ 413us) server_negotiation.cc:408] Connection header received
0203 21:07:26.110226 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.110230 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.110266 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.110323 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:26.111267 (+ 944us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.111954 (+ 687us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.112949 (+ 995us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.113132 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.113974 (+ 842us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:26.113989 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:26.113992 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:26.114014 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:26.115361 (+ 1347us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.115856 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.115859 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.115861 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.115898 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.116087 (+ 189us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.116090 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.116091 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.116243 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:26.116308 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.116583 (+ 275us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.116728 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36,"spinlock_wait_cycles":8064}
I20260203 21:07:26.117082 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.109518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:54468)
0203 21:07:26.109879 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.109894 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.110019 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.110272 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.110276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.110561 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:26.110900 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.110909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.111702 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.111705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.112095 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.112103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.112198 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.112690 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:26.112707 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:26.113066 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:26.115456 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.115459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.115462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.115761 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.116062 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.116066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.116068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.116123 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.116507 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:26.116511 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:26.116610 (+ 99us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.116770 (+ 160us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.116894 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"spinlock_wait_cycles":1920,"thread_start_us":101,"threads_started":1}
I20260203 21:07:26.117404 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.109563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54468 (local address 127.0.102.190:42297)
0203 21:07:26.109686 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.109691 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.109957 (+ 266us) server_negotiation.cc:408] Connection header received
0203 21:07:26.110131 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.110134 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.110177 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.110248 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:26.111029 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.111593 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.112200 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.112394 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.113214 (+ 820us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:26.113231 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:26.113234 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:26.113262 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:26.114716 (+ 1454us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.115861 (+ 1145us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.115865 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.115867 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.115951 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.116229 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.116233 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.116234 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.116393 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:26.116506 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.116833 (+ 327us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.116935 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260203 21:07:26.117583 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.109778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.189:38693 (local address 127.0.0.1:53306)
0203 21:07:26.109996 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.110009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.110108 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.110393 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.110395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.110589 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:26.111162 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.111170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.112059 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.112062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.112842 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.112852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.112974 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.113509 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:26.113520 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:26.113858 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:26.115456 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.115459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.115462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.115761 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.115975 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.115977 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.115979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.116018 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.116359 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:26.116361 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:26.116423 (+ 62us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.117232 (+ 809us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.117343 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
Feb 03 21:07:26 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.190@KRBTEST.COM for oryx/127.0.102.189@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.102.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.102.188@KRBTEST.COM" created.
I20260203 21:07:26.152601 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.121360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.189:38693 (local address 127.0.0.1:53316)
0203 21:07:26.121773 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.121812 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.121945 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.125145 (+ 3200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.125151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.125643 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:26.125942 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.125953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.126888 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.126892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.127449 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.127464 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.127577 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.128190 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:26.128206 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:26.129947 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:26.150756 (+ 20809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.150768 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.150773 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.151045 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.151329 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.151331 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.151332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.151381 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.151869 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:26.151878 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:26.152031 (+ 153us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.152199 (+ 168us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.152402 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":132,"threads_started":1}
I20260203 21:07:26.153399 7705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.124760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53316 (local address 127.0.102.189:38693)
0203 21:07:26.124871 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.124875 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.124888 (+ 13us) server_negotiation.cc:408] Connection header received
0203 21:07:26.124955 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.124958 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.125010 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.125091 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:26.126083 (+ 992us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.126785 (+ 702us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.127562 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.127746 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.130087 (+ 2341us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:26.130108 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:26.130112 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:26.130141 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:26.150549 (+ 20408us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.151180 (+ 631us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.151184 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.151186 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.151247 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.151470 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.151474 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.151476 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.151754 (+ 278us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:26.151848 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.153122 (+ 1274us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.153250 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260203 21:07:26.154870 7162 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.0.102.189:38693: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c31601e9963b484bbb7869e10d137467"
member_type: NON_VOTER
last_known_addr {
host: "127.0.102.189"
port: 38693
}
attrs {
promote: true
}
}
dest_uuid: "61f5416a4138441e8bdfa4f27aeb963d"
cas_config_opid_index: -1
I20260203 21:07:26.155537 7162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: NON_VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.102.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.102.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
I20260203 21:07:26.157418 7730 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c31601e9963b484bbb7869e10d137467 (127.0.102.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: NON_VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: true } } }
I20260203 21:07:26.158181 7676 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:26.159226 7730 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.0.102.189:38693
I20260203 21:07:26.159513 7676 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 FOLLOWER]: Refusing update from remote peer 61f5416a4138441e8bdfa4f27aeb963d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260203 21:07:26.159693 7634 master.cc:561] Master@127.0.102.189:38693 shutting down...
I20260203 21:07:26.160037 7731 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: NON_VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260203 21:07:26.161557 7149 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260203 21:07:26.162257 7149 consensus_peers.cc:597] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d -> Peer c31601e9963b484bbb7869e10d137467 (127.0.102.189:38693): Couldn't send request to peer c31601e9963b484bbb7869e10d137467. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260203 21:07:26.163434 7733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "61f5416a4138441e8bdfa4f27aeb963d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: NON_VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: true } } }
I20260203 21:07:26.163556 7733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:26.164150 7634 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:26.164357 7634 raft_consensus.cc:2272] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:26.164422 7634 tablet_replica.cc:333] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: stopping tablet replica
I20260203 21:07:26.167951 7634 master.cc:583] Master@127.0.102.189:38693 shutdown complete.
I20260203 21:07:26.167991 7634 master_runner.cc:305] Clearing existing system tablet
I20260203 21:07:26.168164 7634 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:26.168207 7634 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:26.169166 7634 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:26.169842 7737 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
WARNING: no policy specified for HTTP/127.0.102.188@KRBTEST.COM; defaulting to no policy
I20260203 21:07:26.170030 7634 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:26.170181 7634 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:26.170217 7634 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/wal
uuid: "c31601e9963b484bbb7869e10d137467"
format_stamp: "Formatted at 2026-02-03 21:07:26 on dist-test-slave-j889"
server_key: "7d6f6d7c96a584bc38de1754ac71b0b4"
server_key_iv: "f48589ed72693a3a5a22dbcc1f211630"
server_key_version: "encryptionkey@0"
I20260203 21:07:26.171631 7634 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.188@KRBTEST.COM" created.
I20260203 21:07:26.173574 7634 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260203 21:07:26.173650 7634 log.cc:1199] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260203 21:07:26.173847 7634 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: Deleting consensus metadata
I20260203 21:07:26.173988 7634 master_runner.cc:315] Copying system tablet from 127.0.102.190:42297
I20260203 21:07:26.174949 7634 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: tablet copy: Beginning tablet copy session from remote peer at address 127.0.102.190:42297
I20260203 21:07:26.181265 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.175456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.0.102.190:42297)
0203 21:07:26.175586 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.175591 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.175774 (+ 183us) server_negotiation.cc:408] Connection header received
0203 21:07:26.175894 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.175897 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.175944 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.176003 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:26.176510 (+ 507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.177080 (+ 570us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.177660 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.177845 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.178659 (+ 814us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:26.178675 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:26.178677 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:26.178699 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:26.179861 (+ 1162us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.180263 (+ 402us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.180266 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.180267 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.180309 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.180508 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.180511 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.180513 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.180700 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:26.180792 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.181027 (+ 235us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.181123 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260203 21:07:26.181281 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.175387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:54474)
0203 21:07:26.175694 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.175705 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.175807 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.176027 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.176029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.176206 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:26.176398 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.176404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.177181 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.177184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.177562 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.177569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.177664 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.178187 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:26.178203 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:26.178552 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:26.179960 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.179964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.179967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.180119 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.180388 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.180391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.180392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.180435 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.180792 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:26.180795 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:26.180877 (+ 82us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.180968 (+ 91us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.181098 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":77,"threads_started":1}
I20260203 21:07:26.182138 7182 tablet_copy_service.cc:140] P 61f5416a4138441e8bdfa4f27aeb963d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c31601e9963b484bbb7869e10d137467 ({username='oryx', principal='oryx/127.0.102.189@KRBTEST.COM'} at 127.0.0.1:54474)
I20260203 21:07:26.182307 7182 tablet_copy_service.cc:161] P 61f5416a4138441e8bdfa4f27aeb963d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c31601e9963b484bbb7869e10d137467 at {username='oryx', principal='oryx/127.0.102.189@KRBTEST.COM'} at 127.0.0.1:54474: session id = c31601e9963b484bbb7869e10d137467-00000000000000000000000000000000
I20260203 21:07:26.183921 7182 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d: Tablet Copy: opened 0 blocks and 1 log segments
I20260203 21:07:26.185593 7634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260203 21:07:26.187686 7634 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: tablet copy: Starting download of 0 data blocks...
I20260203 21:07:26.188136 7634 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: tablet copy: Starting download of 1 WAL segments...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.102.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab.
I20260203 21:07:26.190099 7634 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260203 21:07:26.190076 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.188:45925
--webserver_interface=127.0.102.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:42297,127.0.102.189:38693,127.0.102.188:45925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260203 21:07:26.191881 7182 tablet_copy_service.cc:342] P 61f5416a4138441e8bdfa4f27aeb963d: Request end of tablet copy session c31601e9963b484bbb7869e10d137467-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.102.189@KRBTEST.COM'} at 127.0.0.1:54474
I20260203 21:07:26.191990 7182 tablet_copy_service.cc:434] P 61f5416a4138441e8bdfa4f27aeb963d: ending tablet copy session c31601e9963b484bbb7869e10d137467-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c31601e9963b484bbb7869e10d137467
W20260203 21:07:26.196204 7634 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260203 21:07:26.198360 7634 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:26.199991 7746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:26.200040 7749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:26.200376 7634 server_base.cc:1060] running on GCE node
W20260203 21:07:26.200505 7747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:26.200731 7634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:26.200995 7634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:26.202128 7634 hybrid_clock.cc:648] HybridClock initialized: now 1770152846202118 us; error 25 us; skew 500 ppm
Feb 03 21:07:26 dist-test-slave-j889 krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:26.203456 7634 init.cc:377] Logged in from keytab as oryx/127.0.102.189@KRBTEST.COM (short username oryx)
I20260203 21:07:26.203905 7634 webserver.cc:492] Webserver started at http://127.0.102.189:44503/ using document root <none> and password file <none>
I20260203 21:07:26.204067 7634 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:26.204104 7634 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:26.205148 7634 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260203 21:07:26.205947 7756 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:26.206147 7634 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:26.206210 7634 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/wal
uuid: "c31601e9963b484bbb7869e10d137467"
format_stamp: "Formatted at 2026-02-03 21:07:26 on dist-test-slave-j889"
server_key: "7d6f6d7c96a584bc38de1754ac71b0b4"
server_key_iv: "f48589ed72693a3a5a22dbcc1f211630"
server_key_version: "encryptionkey@0"
I20260203 21:07:26.206285 7634 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:26.213613 7634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:26.215256 7634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:26.219535 7634 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.189:38693
I20260203 21:07:26.219702 7808 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.189:38693 every 8 connection(s)
I20260203 21:07:26.220101 7634 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-1/data/info.pb
I20260203 21:07:26.221552 7809 sys_catalog.cc:263] Verifying existing consensus state
I20260203 21:07:26.222347 7809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: Bootstrap starting.
I20260203 21:07:26.230419 7809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260203 21:07:26.230720 7809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: Bootstrap complete.
I20260203 21:07:26.231197 7809 raft_consensus.cc:359] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: NON_VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: true } }
I20260203 21:07:26.231302 7809 raft_consensus.cc:740] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c31601e9963b484bbb7869e10d137467, State: Initialized, Role: LEARNER
I20260203 21:07:26.231405 7809 consensus_queue.cc:260] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: NON_VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: true } }
I20260203 21:07:26.231742 7812 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: NON_VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: true } } }
I20260203 21:07:26.231878 7809 sys_catalog.cc:565] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:07:26.231889 7812 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: This master's current role is: LEARNER
I20260203 21:07:26.234565 7823 catalog_manager.cc:1269] Loaded cluster ID: 236be3fb335142d6a4092ad9c8733b3a
I20260203 21:07:26.234877 7823 catalog_manager.cc:1562] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: loading cluster ID for follower catalog manager: success
I20260203 21:07:26.236492 7823 catalog_manager.cc:1584] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: acquiring CA information for follower catalog manager: success
I20260203 21:07:26.237597 7823 catalog_manager.cc:1612] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260203 21:07:26.338920 7745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:26.339159 7745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:26.339212 7745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:26.342603 7745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:07:26.342680 7745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:26.342702 7745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:26.342720 7745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:07:26.342737 7745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:07:26.347150 7745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:42297,127.0.102.189:38693,127.0.102.188:45925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.188:45925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.102.188
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:26.348176 7745 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:26.348991 7745 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:26.354244 7828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:26.354400 7745 server_base.cc:1060] running on GCE node
W20260203 21:07:26.354262 7829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:26.354245 7831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:26.355062 7745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:26.355913 7745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:26.357056 7745 hybrid_clock.cc:648] HybridClock initialized: now 1770152846357044 us; error 27 us; skew 500 ppm
Feb 03 21:07:26 dist-test-slave-j889 krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:26.359547 7745 init.cc:377] Logged in from keytab as oryx/127.0.102.188@KRBTEST.COM (short username oryx)
I20260203 21:07:26.360507 7745 webserver.cc:492] Webserver started at http://127.0.102.188:46023/ using document root <none> and password file <none>
I20260203 21:07:26.361034 7745 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:26.361083 7745 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:26.361250 7745 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:26.362829 7745 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/data/instance:
uuid: "0ef1705ecf404c62b8d1ca281e959679"
format_stamp: "Formatted at 2026-02-03 21:07:26 on dist-test-slave-j889"
server_key: "1404d259ea3a14d2b77cdf426ce9236d"
server_key_iv: "f9355ffea3d083bb298e4a475ec28001"
server_key_version: "encryptionkey@0"
I20260203 21:07:26.363252 7745 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/wal/instance:
uuid: "0ef1705ecf404c62b8d1ca281e959679"
format_stamp: "Formatted at 2026-02-03 21:07:26 on dist-test-slave-j889"
server_key: "1404d259ea3a14d2b77cdf426ce9236d"
server_key_iv: "f9355ffea3d083bb298e4a475ec28001"
server_key_version: "encryptionkey@0"
I20260203 21:07:26.366390 7745 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260203 21:07:26.368396 7838 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:26.369335 7745 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260203 21:07:26.369433 7745 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/wal
uuid: "0ef1705ecf404c62b8d1ca281e959679"
format_stamp: "Formatted at 2026-02-03 21:07:26 on dist-test-slave-j889"
server_key: "1404d259ea3a14d2b77cdf426ce9236d"
server_key_iv: "f9355ffea3d083bb298e4a475ec28001"
server_key_version: "encryptionkey@0"
I20260203 21:07:26.369529 7745 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:26.394963 7745 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:26.397960 7745 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:26.398132 7745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:26.405659 7745 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.188:45925
I20260203 21:07:26.405659 7890 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.188:45925 every 8 connection(s)
I20260203 21:07:26.406738 7745 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/data/info.pb
I20260203 21:07:26.409534 7891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:26.413700 7891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } has no permanent_uuid. Determining permanent_uuid...
I20260203 21:07:26.417025 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 7745
Feb 03 21:07:26 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.188@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM
Feb 03 21:07:26 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.102.188@KRBTEST.COM
I20260203 21:07:26.425101 7355 heartbeater.cc:499] Master 127.0.102.190:42297 was elected leader, sending a full tablet report...
I20260203 21:07:26.425774 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.415448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.0.102.190:42297)
0203 21:07:26.415579 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.415583 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.416172 (+ 589us) server_negotiation.cc:408] Connection header received
0203 21:07:26.416985 (+ 813us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.416989 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.417040 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.417121 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:26.418581 (+ 1460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.419070 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.419661 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.419844 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.422097 (+ 2253us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:26.422118 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:26.422120 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:26.422149 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:26.423584 (+ 1435us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.424216 (+ 632us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.424219 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.424221 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.424269 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.424553 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.424555 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.424556 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.424743 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:26.424818 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.425475 (+ 657us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.425627 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260203 21:07:26.426524 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.415574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:54482)
0203 21:07:26.416025 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.416066 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.416757 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.417308 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.417320 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.417746 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:26.418428 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.418441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.419178 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.419181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.419557 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.419563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.419747 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.420321 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:26.420347 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:26.421939 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:26.423717 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.423730 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.423741 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.424120 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.424347 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.424350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.424352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.424455 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.424840 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:26.424848 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:26.425154 (+ 306us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.425412 (+ 258us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.425698 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":135,"threads_started":1}
I20260203 21:07:26.426806 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.417861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.0.102.188:45925)
0203 21:07:26.418164 (+ 303us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.418169 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.418187 (+ 18us) server_negotiation.cc:408] Connection header received
0203 21:07:26.418219 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.418223 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.418364 (+ 141us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.418503 (+ 139us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:26.419271 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.419935 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.420523 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.420772 (+ 249us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.422858 (+ 2086us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:26.422881 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:26.422981 (+ 100us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:26.423009 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:26.424930 (+ 1921us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.425359 (+ 429us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.425365 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.425366 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.425416 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.425698 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.425701 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.425703 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.426153 (+ 450us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:26.426343 (+ 190us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.426528 (+ 185us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.426667 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":119,"threads_started":1}
I20260203 21:07:26.427796 7891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } has no permanent_uuid. Determining permanent_uuid...
Feb 03 21:07:26 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.188@KRBTEST.COM for oryx/127.0.102.189@KRBTEST.COM
I20260203 21:07:26.436473 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.428242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.189:38693 (local address 127.0.0.1:53320)
0203 21:07:26.428351 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.428364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.428423 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.428898 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.428901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.429027 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:26.429199 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.429204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.430298 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.430301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.430699 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.430706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.430796 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.431262 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:26.431278 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:26.432705 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:26.434755 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.434759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.434761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.435072 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.435351 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.435354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.435356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.435439 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.435809 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:26.435812 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:26.435951 (+ 139us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.436069 (+ 118us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.436243 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260203 21:07:26.437397 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.428367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.0.102.189:38693)
0203 21:07:26.428676 (+ 309us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.428680 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.428692 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:26.428731 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.428734 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.428781 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.428853 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:26.429295 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.430192 (+ 897us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.430795 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.430936 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.432836 (+ 1900us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:26.432857 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:26.432861 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:26.432888 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:26.434628 (+ 1740us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.435193 (+ 565us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.435197 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.435199 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.435256 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.435540 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.435543 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.435545 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.435726 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:26.435830 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.436997 (+ 1167us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.437170 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":98,"threads_started":1}
I20260203 21:07:26.441262 7891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } has no permanent_uuid. Determining permanent_uuid...
Feb 03 21:07:26 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.188@KRBTEST.COM for oryx/127.0.102.188@KRBTEST.COM
I20260203 21:07:26.449337 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.441903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.188:45925 (local address 127.0.0.1:55130)
0203 21:07:26.442138 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.442152 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.442239 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.442503 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.442506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.442715 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:26.442956 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.442963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.443880 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.443883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.444212 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.444218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.444315 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.445023 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:26.445037 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:26.446217 (+ 1180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:26.447947 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.447950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.447951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.448119 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.448391 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.448394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.448395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.448436 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.448874 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:26.448877 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:26.448961 (+ 84us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.449066 (+ 105us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.449208 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260203 21:07:26.449448 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.442164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.0.102.188:45925)
0203 21:07:26.442295 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.442299 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.442311 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:26.442347 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.442350 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.442394 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.442480 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:26.443081 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.443775 (+ 694us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.444300 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.444503 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.446325 (+ 1822us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:26.446343 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:26.446345 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:26.446366 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:26.447833 (+ 1467us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.448251 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.448254 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.448255 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.448305 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.448539 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.448543 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.448546 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.448761 (+ 215us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:26.448861 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.449108 (+ 247us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.449289 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":61}
I20260203 21:07:26.451988 7891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: Bootstrap starting.
I20260203 21:07:26.454420 7891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:26.455183 7891 log.cc:826] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:26.457098 7891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: No bootstrap required, opened a new log
I20260203 21:07:26.459569 7891 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } }
I20260203 21:07:26.459748 7891 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:26.459784 7891 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ef1705ecf404c62b8d1ca281e959679, State: Initialized, Role: FOLLOWER
I20260203 21:07:26.460211 7891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } }
I20260203 21:07:26.460808 7897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } } }
I20260203 21:07:26.460923 7897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [sys.catalog]: This master's current role is: FOLLOWER
I20260203 21:07:26.461373 7891 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [sys.catalog]: configured and running, proceeding with master startup.
W20260203 21:07:26.466694 7908 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260203 21:07:26.466799 7908 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260203 21:07:26.469740 7745 master_runner.cc:428] Detected that this master 0ef1705ecf404c62b8d1ca281e959679 is joining an existing cluster
I20260203 21:07:26.469813 7745 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.102.188:45925
I20260203 21:07:26.483335 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.480835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:54504)
0203 21:07:26.480944 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.480959 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.481035 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.481400 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.481404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.481419 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:26.481589 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.481594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.482305 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.482308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.482895 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.482902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.483008 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.483077 (+ 69us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.483131 (+ 54us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.483195 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260203 21:07:26.483691 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.480913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.0.102.190:42297)
0203 21:07:26.481217 (+ 304us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.481220 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.481230 (+ 10us) server_negotiation.cc:408] Connection header received
0203 21:07:26.481270 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.481273 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.481308 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.481378 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:26.481694 (+ 316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.482204 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.483004 (+ 800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.483417 (+ 413us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.483490 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.483538 (+ 48us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.483603 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":123,"threads_started":1}
I20260203 21:07:26.487207 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.484841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.189:38693 (local address 127.0.0.1:53336)
0203 21:07:26.484950 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.484962 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.485021 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.485436 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.485439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.485456 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:26.485611 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.485616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.486307 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.486312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.486870 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.486876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.486969 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.486986 (+ 17us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.487038 (+ 52us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.487093 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260203 21:07:26.487726 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.484959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.0.102.189:38693)
0203 21:07:26.485229 (+ 270us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.485234 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.485245 (+ 11us) server_negotiation.cc:408] Connection header received
0203 21:07:26.485281 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.485284 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.485331 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.485414 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:26.485713 (+ 299us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.486206 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.486974 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.487419 (+ 445us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.487549 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.487593 (+ 44us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.487637 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
I20260203 21:07:26.496176 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.472411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.188:45925 (local address 127.0.0.1:55140)
0203 21:07:26.472742 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.472755 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.472835 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.473006 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.473009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.488930 (+ 15921us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:26.489665 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.489675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.490474 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.490477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.490942 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.490951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.491042 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.491592 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:26.491606 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:26.492007 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:26.494877 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.494880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.494882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.495076 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.495281 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.495284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.495286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.495327 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.495664 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:26.495667 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:26.495748 (+ 81us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.495856 (+ 108us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.495979 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"spinlock_wait_cycles":3840,"thread_start_us":78,"threads_started":1}
I20260203 21:07:26.496676 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.472260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.0.102.188:45925)
0203 21:07:26.472354 (+ 94us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.472358 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.472803 (+ 445us) server_negotiation.cc:408] Connection header received
0203 21:07:26.472895 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.472899 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.472930 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.472984 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:26.489792 (+ 16808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.490358 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.491271 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.491436 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.493230 (+ 1794us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:26.493252 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:26.493255 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:26.493281 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:26.494768 (+ 1487us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.495168 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.495171 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.495173 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.495211 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.495395 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.495398 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.495401 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.495558 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:26.495635 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.496405 (+ 770us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.496506 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38,"spinlock_wait_cycles":11136}
I20260203 21:07:26.496909 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.471806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53328 (local address 127.0.102.189:38693)
0203 21:07:26.471922 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.471926 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.472174 (+ 248us) server_negotiation.cc:408] Connection header received
0203 21:07:26.472355 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.472359 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.472415 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.472468 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:26.490283 (+ 17815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.491036 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.491853 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.492037 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.493289 (+ 1252us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:26.493308 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:26.493312 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:26.493340 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:26.495001 (+ 1661us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.495548 (+ 547us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.495552 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.495553 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.495602 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.495868 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.495871 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.495873 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.496038 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:26.496116 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.496602 (+ 486us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.496765 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260203 21:07:26.497296 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.471812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.189:38693 (local address 127.0.0.1:53328)
0203 21:07:26.472099 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.472113 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.472245 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.472609 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.472612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.489890 (+ 17278us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:26.490150 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.490157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.491168 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.491171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.491727 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.491737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.492144 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.492624 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:26.492677 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:26.493145 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:26.495126 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.495130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.495132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.495419 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.495710 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.495712 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.495714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.495765 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.496183 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:26.496186 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:26.496407 (+ 221us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.497032 (+ 625us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.497152 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"spinlock_wait_cycles":1408,"thread_start_us":122,"threads_started":1}
I20260203 21:07:26.499224 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.471735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.0.102.190:42297)
0203 21:07:26.471862 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.471866 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.472281 (+ 415us) server_negotiation.cc:408] Connection header received
0203 21:07:26.472447 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.472451 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.472503 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.472563 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:26.489360 (+ 16797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.489837 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.491975 (+ 2138us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.492141 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.493389 (+ 1248us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:26.493403 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:26.493406 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:26.493430 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:26.495021 (+ 1591us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.498300 (+ 3279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.498303 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.498304 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.498367 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.498584 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.498586 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.498587 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.498718 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:26.498796 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.499030 (+ 234us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.499122 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260203 21:07:26.499284 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.471897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:54492)
0203 21:07:26.472200 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.472214 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.472346 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.472737 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.472740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.488949 (+ 16209us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:26.489238 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.489247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.489950 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.489953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.491839 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.491849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.492262 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.492743 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:26.492756 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:26.493145 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:26.495126 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.495130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.495132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.498198 (+ 3066us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.498455 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.498458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.498460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.498512 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.498802 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:26.498806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:26.498897 (+ 91us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.498994 (+ 97us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.499109 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"spinlock_wait_cycles":3840,"thread_start_us":103,"threads_started":1}
Feb 03 21:07:26 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152845, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.190@KRBTEST.COM for oryx/127.0.102.188@KRBTEST.COM
I20260203 21:07:26.511668 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.502765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55150 (local address 127.0.102.188:45925)
0203 21:07:26.502883 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.502887 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.503388 (+ 501us) server_negotiation.cc:408] Connection header received
0203 21:07:26.503480 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.503484 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.503542 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.503644 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:26.504306 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.504958 (+ 652us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.505685 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.505871 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.508093 (+ 2222us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:26.508116 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:26.508118 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:26.508149 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:26.509909 (+ 1760us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.510449 (+ 540us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.510453 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.510454 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.510506 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.510775 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.510778 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.510779 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.511019 (+ 240us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:26.511117 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.511410 (+ 293us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.511520 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260203 21:07:26.511668 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.502697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.188:45925 (local address 127.0.0.1:55150)
0203 21:07:26.503215 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.503230 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.503320 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.503773 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.503776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.503963 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:26.504201 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.504211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.505150 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.505153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.505580 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.505588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.505691 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.506287 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:26.506302 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:26.507910 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:26.510055 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.510060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.510063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.510313 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.510597 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.510600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.510603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.510659 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.511115 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:26.511118 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:26.511221 (+ 103us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.511328 (+ 107us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.511487 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438}
I20260203 21:07:26.512737 7162 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.0.102.188:45925: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679"
member_type: NON_VOTER
last_known_addr {
host: "127.0.102.188"
port: 45925
}
attrs {
promote: true
}
}
dest_uuid: "61f5416a4138441e8bdfa4f27aeb963d"
cas_config_opid_index: 5
I20260203 21:07:26.513270 7162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: NON_VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: true } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } }
I20260203 21:07:26.514294 7731 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0ef1705ecf404c62b8d1ca281e959679 (127.0.102.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: NON_VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: true } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } } }
I20260203 21:07:26.514847 7865 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:26.515703 7731 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.0.102.188:45925
I20260203 21:07:26.515895 7731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "61f5416a4138441e8bdfa4f27aeb963d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: NON_VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: true } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } } }
I20260203 21:07:26.515993 7731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:26.516006 7865 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 1 FOLLOWER]: Refusing update from remote peer 61f5416a4138441e8bdfa4f27aeb963d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260203 21:07:26.516494 7745 master.cc:561] Master@127.0.102.188:45925 shutting down...
I20260203 21:07:26.516539 7730 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:26.517024 7783 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 LEARNER]: Refusing update from remote peer 61f5416a4138441e8bdfa4f27aeb963d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260203 21:07:26.517828 7730 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: NON_VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260203 21:07:26.519110 7147 consensus_peers.cc:597] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d -> Peer 0ef1705ecf404c62b8d1ca281e959679 (127.0.102.188:45925): Couldn't send request to peer 0ef1705ecf404c62b8d1ca281e959679. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260203 21:07:26.519992 7783 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0ef1705ecf404c62b8d1ca281e959679 (127.0.102.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: NON_VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: true } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } } }
I20260203 21:07:26.521765 7920 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61f5416a4138441e8bdfa4f27aeb963d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: NON_VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: true } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } } }
I20260203 21:07:26.521955 7920 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: This master's current role is: LEARNER
I20260203 21:07:26.522620 7812 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61f5416a4138441e8bdfa4f27aeb963d. Latest consensus state: current_term: 1 leader_uuid: "61f5416a4138441e8bdfa4f27aeb963d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: NON_VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: true } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } } }
I20260203 21:07:26.522727 7812 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: This master's current role is: LEARNER
I20260203 21:07:26.523437 7745 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:26.523615 7730 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d: attempting to promote NON_VOTER c31601e9963b484bbb7869e10d137467 to VOTER
I20260203 21:07:26.523705 7745 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:26.523777 7745 tablet_replica.cc:333] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: stopping tablet replica
I20260203 21:07:26.524052 7730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } }
W20260203 21:07:26.525890 7147 consensus_peers.cc:597] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d -> Peer 0ef1705ecf404c62b8d1ca281e959679 (127.0.102.188:45925): Couldn't send request to peer 0ef1705ecf404c62b8d1ca281e959679. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260203 21:07:26.526113 7783 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 LEARNER]: Refusing update from remote peer 61f5416a4138441e8bdfa4f27aeb963d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260203 21:07:26.526490 7733 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260203 21:07:26.528254 7745 master.cc:583] Master@127.0.102.188:45925 shutdown complete.
I20260203 21:07:26.528316 7745 master_runner.cc:305] Clearing existing system tablet
I20260203 21:07:26.528558 7745 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:26.528614 7745 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:26.529939 7745 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:26.530210 7731 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c31601e9963b484bbb7869e10d137467 (127.0.102.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } } }
I20260203 21:07:26.530819 7783 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c31601e9963b484bbb7869e10d137467 (127.0.102.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } } }
I20260203 21:07:26.531998 7731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61f5416a4138441e8bdfa4f27aeb963d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } } }
I20260203 21:07:26.532100 7731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:26.532128 7812 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61f5416a4138441e8bdfa4f27aeb963d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } } }
I20260203 21:07:26.532243 7812 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: This master's current role is: FOLLOWER
I20260203 21:07:26.532349 7730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "61f5416a4138441e8bdfa4f27aeb963d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } } }
I20260203 21:07:26.532445 7730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:26.532830 7922 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:26.533068 7745 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:26.533193 7745 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260203 21:07:26.533236 7745 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/wal
uuid: "0ef1705ecf404c62b8d1ca281e959679"
format_stamp: "Formatted at 2026-02-03 21:07:26 on dist-test-slave-j889"
server_key: "1404d259ea3a14d2b77cdf426ce9236d"
server_key_iv: "f9355ffea3d083bb298e4a475ec28001"
server_key_version: "encryptionkey@0"
I20260203 21:07:26.535192 7745 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:26.537012 7745 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260203 21:07:26.537093 7745 log.cc:1199] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260203 21:07:26.537297 7745 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: Deleting consensus metadata
I20260203 21:07:26.537431 7745 master_runner.cc:315] Copying system tablet from 127.0.102.190:42297
I20260203 21:07:26.538511 7745 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: tablet copy: Beginning tablet copy session from remote peer at address 127.0.102.190:42297
I20260203 21:07:26.542974 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.542609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.188:45925 (local address 127.0.0.1:55164)
0203 21:07:26.542808 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.542879 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.188:45925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260203 21:07:26.543249 7161 master.cc:679] Network error: unable to get registration information for peer 0ef1705ecf404c62b8d1ca281e959679: Client connection negotiation failed: client connection to 127.0.102.188:45925: connect: Connection refused (error 111)
I20260203 21:07:26.546397 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.539112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.0.102.190:42297)
0203 21:07:26.539232 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.539237 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.539469 (+ 232us) server_negotiation.cc:408] Connection header received
0203 21:07:26.539682 (+ 213us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.539686 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.539737 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.539791 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:26.540494 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.541257 (+ 763us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.542050 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.542223 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.543141 (+ 918us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:26.543159 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:26.543162 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:26.543195 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:26.544801 (+ 1606us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.545163 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.545167 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.545169 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.545216 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:26.545410 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:26.545414 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:26.545416 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:26.545684 (+ 268us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:26.545802 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.546103 (+ 301us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.546211 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260203 21:07:26.546416 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.539024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:54512)
0203 21:07:26.539375 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.539390 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.539540 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.539922 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.539925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.540111 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:26.540368 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.540375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.541406 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.541409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.541949 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.541958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.542072 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.542646 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:26.542659 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:26.543004 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:26.544903 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.544906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.544909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.545076 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.545289 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:26.545292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:26.545294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:26.545338 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:26.545835 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:26.545839 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:26.545933 (+ 94us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.546036 (+ 103us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.546200 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":98,"threads_started":1}
I20260203 21:07:26.547202 7182 tablet_copy_service.cc:140] P 61f5416a4138441e8bdfa4f27aeb963d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0ef1705ecf404c62b8d1ca281e959679 ({username='oryx', principal='oryx/127.0.102.188@KRBTEST.COM'} at 127.0.0.1:54512)
I20260203 21:07:26.547298 7182 tablet_copy_service.cc:161] P 61f5416a4138441e8bdfa4f27aeb963d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0ef1705ecf404c62b8d1ca281e959679 at {username='oryx', principal='oryx/127.0.102.188@KRBTEST.COM'} at 127.0.0.1:54512: session id = 0ef1705ecf404c62b8d1ca281e959679-00000000000000000000000000000000
I20260203 21:07:26.548075 7182 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d: Tablet Copy: opened 0 blocks and 1 log segments
I20260203 21:07:26.549266 7745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260203 21:07:26.551016 7745 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: tablet copy: Starting download of 0 data blocks...
I20260203 21:07:26.551172 7745 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: tablet copy: Starting download of 1 WAL segments...
I20260203 21:07:26.552991 7745 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260203 21:07:26.554666 7182 tablet_copy_service.cc:342] P 61f5416a4138441e8bdfa4f27aeb963d: Request end of tablet copy session 0ef1705ecf404c62b8d1ca281e959679-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.102.188@KRBTEST.COM'} at 127.0.0.1:54512
I20260203 21:07:26.554749 7182 tablet_copy_service.cc:434] P 61f5416a4138441e8bdfa4f27aeb963d: ending tablet copy session 0ef1705ecf404c62b8d1ca281e959679-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0ef1705ecf404c62b8d1ca281e959679
W20260203 21:07:26.559443 7745 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260203 21:07:26.561708 7745 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:26.563468 7929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:26.563476 7932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:26.563613 7930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:26.563917 7745 server_base.cc:1060] running on GCE node
I20260203 21:07:26.564086 7745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:26.564354 7745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:26.565481 7745 hybrid_clock.cc:648] HybridClock initialized: now 1770152846565473 us; error 24 us; skew 500 ppm
Feb 03 21:07:26 dist-test-slave-j889 krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152846, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:26.567023 7745 init.cc:377] Logged in from keytab as oryx/127.0.102.188@KRBTEST.COM (short username oryx)
I20260203 21:07:26.567539 7745 webserver.cc:492] Webserver started at http://127.0.102.188:45319/ using document root <none> and password file <none>
I20260203 21:07:26.567749 7745 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:26.567788 7745 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:26.569114 7745 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:26.570068 7939 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:26.570397 7745 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:26.570485 7745 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/wal
uuid: "0ef1705ecf404c62b8d1ca281e959679"
format_stamp: "Formatted at 2026-02-03 21:07:26 on dist-test-slave-j889"
server_key: "1404d259ea3a14d2b77cdf426ce9236d"
server_key_iv: "f9355ffea3d083bb298e4a475ec28001"
server_key_version: "encryptionkey@0"
I20260203 21:07:26.570602 7745 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:26.580169 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.579894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.188:45925 (local address 127.0.0.1:55168)
0203 21:07:26.580029 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.580089 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.188:45925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260203 21:07:26.580381 7161 master.cc:679] Network error: unable to get registration information for peer 0ef1705ecf404c62b8d1ca281e959679: Client connection negotiation failed: client connection to 127.0.102.188:45925: connect: Connection refused (error 111)
I20260203 21:07:26.596271 7745 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:26.598500 7745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:26.603121 7745 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.188:45925
I20260203 21:07:26.603286 7991 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.188:45925 every 8 connection(s)
I20260203 21:07:26.603711 7745 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/master-2/data/info.pb
I20260203 21:07:26.605279 7992 sys_catalog.cc:263] Verifying existing consensus state
I20260203 21:07:26.606040 7992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: Bootstrap starting.
I20260203 21:07:26.614894 7992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260203 21:07:26.615166 7992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: Bootstrap complete.
I20260203 21:07:26.615687 7992 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } }
I20260203 21:07:26.615795 7992 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ef1705ecf404c62b8d1ca281e959679, State: Initialized, Role: LEARNER
I20260203 21:07:26.615881 7992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } }
I20260203 21:07:26.616200 7995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } } }
I20260203 21:07:26.616309 7995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [sys.catalog]: This master's current role is: LEARNER
I20260203 21:07:26.616341 7992 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:07:26.619299 8006 catalog_manager.cc:1269] Loaded cluster ID: 236be3fb335142d6a4092ad9c8733b3a
I20260203 21:07:26.619360 8006 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: loading cluster ID for follower catalog manager: success
I20260203 21:07:26.621822 8006 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: acquiring CA information for follower catalog manager: success
I20260203 21:07:26.623150 8006 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260203 21:07:26.645133 7491 heartbeater.cc:499] Master 127.0.102.190:42297 was elected leader, sending a full tablet report...
I20260203 21:07:26.650836 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.648056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.188:45925 (local address 127.0.0.1:55184)
0203 21:07:26.648217 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:26.648229 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:26.648297 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:26.648690 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:26.648693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:26.648714 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:26.648894 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.648901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.649822 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.649826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:26.650415 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:26.650423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.650530 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.650556 (+ 26us) client_negotiation.cc:769] Sending connection context
0203 21:07:26.650614 (+ 58us) client_negotiation.cc:240] Negotiation successful
0203 21:07:26.650682 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260203 21:07:26.651348 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:26.648169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55184 (local address 127.0.102.188:45925)
0203 21:07:26.648440 (+ 271us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:26.648444 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:26.648455 (+ 11us) server_negotiation.cc:408] Connection header received
0203 21:07:26.648496 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:26.648498 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:26.648546 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:26.648669 (+ 123us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:26.649003 (+ 334us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.649707 (+ 704us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:26.650540 (+ 833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:26.651001 (+ 461us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:26.651145 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:26.651190 (+ 45us) server_negotiation.cc:300] Negotiation successful
0203 21:07:26.651246 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
I20260203 21:07:26.856470 7627 heartbeater.cc:499] Master 127.0.102.190:42297 was elected leader, sending a full tablet report...
I20260203 21:07:27.023527 7966 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260203 21:07:27.023960 7995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61f5416a4138441e8bdfa4f27aeb963d. Latest consensus state: current_term: 1 leader_uuid: "61f5416a4138441e8bdfa4f27aeb963d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } } }
I20260203 21:07:27.024109 7995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [sys.catalog]: This master's current role is: LEARNER
I20260203 21:07:27.570154 7730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61f5416a4138441e8bdfa4f27aeb963d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: NON_VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: true } } }
I20260203 21:07:27.570317 7730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:27.571090 7730 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d: attempting to promote NON_VOTER 0ef1705ecf404c62b8d1ca281e959679 to VOTER
I20260203 21:07:27.571465 7730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } }
I20260203 21:07:27.572659 7783 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 FOLLOWER]: Refusing update from remote peer 61f5416a4138441e8bdfa4f27aeb963d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260203 21:07:27.572674 7966 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 1 LEARNER]: Refusing update from remote peer 61f5416a4138441e8bdfa4f27aeb963d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260203 21:07:27.573083 8009 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260203 21:07:27.573272 8008 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260203 21:07:27.574879 8008 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0ef1705ecf404c62b8d1ca281e959679 (127.0.102.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } } }
I20260203 21:07:27.575256 7783 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0ef1705ecf404c62b8d1ca281e959679 (127.0.102.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } } }
I20260203 21:07:27.576234 8008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "61f5416a4138441e8bdfa4f27aeb963d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } } }
I20260203 21:07:27.576331 8008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:27.576468 8008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61f5416a4138441e8bdfa4f27aeb963d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } } }
I20260203 21:07:27.576224 7966 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0ef1705ecf404c62b8d1ca281e959679 (127.0.102.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } } }
I20260203 21:07:27.576498 7730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61f5416a4138441e8bdfa4f27aeb963d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } } }
I20260203 21:07:27.576541 8008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:27.576563 7730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:27.576798 8014 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61f5416a4138441e8bdfa4f27aeb963d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } } }
I20260203 21:07:27.576900 8014 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: This master's current role is: FOLLOWER
I20260203 21:07:27.577756 8015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61f5416a4138441e8bdfa4f27aeb963d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } } }
I20260203 21:07:27.577860 8015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [sys.catalog]: This master's current role is: FOLLOWER
I20260203 21:07:28.575091 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.567904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.0.102.189:38693)
0203 21:07:28.568113 (+ 209us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:28.568117 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:28.568148 (+ 31us) server_negotiation.cc:408] Connection header received
0203 21:07:28.568252 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:28.568255 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:28.568297 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:28.568367 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:28.569223 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.569769 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.570479 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.570629 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.572103 (+ 1474us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:28.572124 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:28.572126 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:28.572153 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:28.573669 (+ 1516us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:28.574047 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:28.574051 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:28.574052 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:28.574106 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:28.574305 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:28.574308 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:28.574311 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:28.574468 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:28.574589 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:28.574787 (+ 198us) server_negotiation.cc:300] Negotiation successful
0203 21:07:28.574914 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":57,"threads_started":1}
I20260203 21:07:28.578857 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.575888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:54514)
0203 21:07:28.576289 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:28.576306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:28.576407 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:28.576729 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:28.576731 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:28.576750 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:28.576947 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.576952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.577784 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.577788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:28.578482 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.578491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.578592 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.578608 (+ 16us) client_negotiation.cc:769] Sending connection context
0203 21:07:28.578651 (+ 43us) client_negotiation.cc:240] Negotiation successful
0203 21:07:28.578710 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":88,"threads_started":1}
I20260203 21:07:28.579383 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.576020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.0.102.190:42297)
0203 21:07:28.576262 (+ 242us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:28.576265 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:28.576365 (+ 100us) server_negotiation.cc:408] Connection header received
0203 21:07:28.576546 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:28.576549 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:28.576601 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:28.576721 (+ 120us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:28.577103 (+ 382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.577656 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.578705 (+ 1049us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.579131 (+ 426us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.579170 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:28.579214 (+ 44us) server_negotiation.cc:300] Negotiation successful
0203 21:07:28.579269 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":95,"threads_started":1}
I20260203 21:07:28.582904 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.580584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.189:38693 (local address 127.0.0.1:53362)
0203 21:07:28.580715 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:28.580728 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:28.580803 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:28.581051 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:28.581053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:28.581071 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:28.581220 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.581225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.581992 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.581996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:28.582603 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.582611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.582701 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.582716 (+ 15us) client_negotiation.cc:769] Sending connection context
0203 21:07:28.582748 (+ 32us) client_negotiation.cc:240] Negotiation successful
0203 21:07:28.582797 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260203 21:07:28.583312 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.580742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.0.102.189:38693)
0203 21:07:28.580875 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:28.580878 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:28.580890 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:28.580927 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:28.580929 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:28.580966 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:28.581027 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:28.581297 (+ 270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.581861 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.582714 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.583103 (+ 389us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.583138 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:28.583179 (+ 41us) server_negotiation.cc:300] Negotiation successful
0203 21:07:28.583225 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260203 21:07:28.586796 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.584419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.188:45925 (local address 127.0.0.1:55186)
0203 21:07:28.584535 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:28.584548 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:28.584611 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:28.585008 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:28.585011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:28.585029 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:28.585180 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.585185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.585927 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.585931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:28.586496 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.586503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.586598 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.586615 (+ 17us) client_negotiation.cc:769] Sending connection context
0203 21:07:28.586650 (+ 35us) client_negotiation.cc:240] Negotiation successful
0203 21:07:28.586694 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260203 21:07:28.587355 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.584542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.0.102.188:45925)
0203 21:07:28.584791 (+ 249us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:28.584819 (+ 28us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:28.584832 (+ 13us) server_negotiation.cc:408] Connection header received
0203 21:07:28.584875 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:28.584877 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:28.584919 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:28.584991 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:28.585282 (+ 291us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.585822 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.586638 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.587123 (+ 485us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.587157 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:28.587204 (+ 47us) server_negotiation.cc:300] Negotiation successful
0203 21:07:28.587251 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":109,"threads_started":1}
I20260203 21:07:28.595415 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.589362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.0.102.188:45925)
0203 21:07:28.589472 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:28.589475 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:28.589486 (+ 11us) server_negotiation.cc:408] Connection header received
0203 21:07:28.589516 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:28.589518 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:28.589556 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:28.589617 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:28.590309 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.590772 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.591323 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.591494 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.592503 (+ 1009us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:28.592521 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:28.592523 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:28.592548 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:28.594187 (+ 1639us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:28.594544 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:28.594546 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:28.594548 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:28.594586 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:28.594753 (+ 167us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:28.594755 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:28.594756 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:28.594924 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:28.595026 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:28.595189 (+ 163us) server_negotiation.cc:300] Negotiation successful
0203 21:07:28.595296 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260203 21:07:28.598690 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.596190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:54524)
0203 21:07:28.596449 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:28.596462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:28.596551 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:28.596787 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:28.596790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:28.596806 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:28.596993 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.596998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.597704 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.597708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:28.598286 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.598293 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.598392 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.598410 (+ 18us) client_negotiation.cc:769] Sending connection context
0203 21:07:28.598457 (+ 47us) client_negotiation.cc:240] Negotiation successful
0203 21:07:28.598512 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":67,"threads_started":1}
I20260203 21:07:28.599061 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.596293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54524 (local address 127.0.102.190:42297)
0203 21:07:28.596444 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:28.596448 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:28.596522 (+ 74us) server_negotiation.cc:408] Connection header received
0203 21:07:28.596664 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:28.596667 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:28.596708 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:28.596768 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:28.597092 (+ 324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.597607 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.598394 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.598866 (+ 472us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.598897 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:28.598937 (+ 40us) server_negotiation.cc:300] Negotiation successful
0203 21:07:28.598978 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260203 21:07:28.602564 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.600223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.189:38693 (local address 127.0.0.1:53378)
0203 21:07:28.600354 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:28.600366 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:28.600431 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:28.600702 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:28.600705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:28.600738 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:28.600943 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.600950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.601645 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.601648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:28.602277 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.602288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.602379 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.602393 (+ 14us) client_negotiation.cc:769] Sending connection context
0203 21:07:28.602425 (+ 32us) client_negotiation.cc:240] Negotiation successful
0203 21:07:28.602472 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260203 21:07:28.603000 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.600353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.0.102.189:38693)
0203 21:07:28.600474 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:28.600477 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:28.600489 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:28.600529 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:28.600532 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:28.600571 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:28.600659 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:28.601065 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.601539 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.602401 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.602768 (+ 367us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.602809 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:28.602859 (+ 50us) server_negotiation.cc:300] Negotiation successful
0203 21:07:28.602912 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260203 21:07:28.606364 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.604187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.188:45925 (local address 127.0.0.1:55216)
0203 21:07:28.604325 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:28.604341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:28.604443 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:28.604654 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:28.604657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:28.604672 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:28.604820 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.604825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.605462 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.605465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:28.606032 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.606039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.606134 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.606152 (+ 18us) client_negotiation.cc:769] Sending connection context
0203 21:07:28.606198 (+ 46us) client_negotiation.cc:240] Negotiation successful
0203 21:07:28.606251 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260203 21:07:28.606704 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.604298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.0.102.188:45925)
0203 21:07:28.604405 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:28.604409 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:28.604421 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:28.604524 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:28.604527 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:28.604562 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:28.604619 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:28.604907 (+ 288us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.605372 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.606132 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.606530 (+ 398us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.606559 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:28.606595 (+ 36us) server_negotiation.cc:300] Negotiation successful
0203 21:07:28.606635 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260203 21:07:28.608157 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 7226
I20260203 21:07:28.613623 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:36543
--local_ip_for_outbound_sockets=127.0.102.129
--tserver_master_addrs=127.0.102.190:42297
--webserver_port=34797
--webserver_interface=127.0.102.129
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.102.190:42297,127.0.102.189:38693
--tserver_master_addrs=127.0.102.190:42297,127.0.102.189:38693,127.0.102.188:45925 with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:28.720726 8022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:28.720949 8022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:28.721014 8022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:28.724351 8022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:28.724419 8022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:28.724509 8022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:07:28.728734 8022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:36543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=34797
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:42297,127.0.102.189:38693,127.0.102.188:45925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:28.729768 8022 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:28.730619 8022 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:28.737313 8027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:28.737313 8028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:28.737371 8030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:28.737360 8022 server_base.cc:1060] running on GCE node
I20260203 21:07:28.737874 8022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:28.738435 8022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:28.739590 8022 hybrid_clock.cc:648] HybridClock initialized: now 1770152848739565 us; error 39 us; skew 500 ppm
Feb 03 21:07:28 dist-test-slave-j889 krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:28.742342 8022 init.cc:377] Logged in from keytab as oryx/127.0.102.129@KRBTEST.COM (short username oryx)
I20260203 21:07:28.743443 8022 webserver.cc:492] Webserver started at http://127.0.102.129:34797/ using document root <none> and password file <none>
I20260203 21:07:28.743997 8022 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:28.744064 8022 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:28.747709 8022 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260203 21:07:28.749651 8037 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:28.750687 8022 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:28.750795 8022 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "f1f9e3fc5a4e44edaca392a6e083f178"
format_stamp: "Formatted at 2026-02-03 21:07:25 on dist-test-slave-j889"
server_key: "1583637660e9c74e2a3d474b34e41cc1"
server_key_iv: "5f684fc980a1aee2b88cf587d36ac460"
server_key_version: "encryptionkey@0"
I20260203 21:07:28.751223 8022 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:28.762706 8022 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:28.765307 8022 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:28.765476 8022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:28.766019 8022 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:28.766898 8022 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:28.766947 8022 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:28.766999 8022 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:28.767022 8022 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:28.775755 8022 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:36543
I20260203 21:07:28.775775 8150 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:36543 every 8 connection(s)
I20260203 21:07:28.776808 8022 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:07:28.779631 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 8022
I20260203 21:07:28.779824 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 7362
I20260203 21:07:28.787087 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:45287
--local_ip_for_outbound_sockets=127.0.102.130
--tserver_master_addrs=127.0.102.190:42297
--webserver_port=37805
--webserver_interface=127.0.102.130
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.102.190:42297,127.0.102.189:38693
--tserver_master_addrs=127.0.102.190:42297,127.0.102.189:38693,127.0.102.188:45925 with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Feb 03 21:07:28 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM
Feb 03 21:07:28 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.188@KRBTEST.COM
Feb 03 21:07:28 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.189@KRBTEST.COM
I20260203 21:07:28.793058 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.778496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:37471 (local address 127.0.102.190:42297)
0203 21:07:28.778641 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:28.778646 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:28.779380 (+ 734us) server_negotiation.cc:408] Connection header received
0203 21:07:28.781536 (+ 2156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:28.781541 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:28.781597 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:28.781672 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:28.783089 (+ 1417us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.783881 (+ 792us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.784819 (+ 938us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.785027 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.788872 (+ 3845us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:28.788899 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:28.788902 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:28.788937 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:28.790538 (+ 1601us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:28.791357 (+ 819us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:28.791361 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:28.791363 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:28.791420 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:28.791743 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:28.791747 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:28.791749 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:28.791943 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:28.792010 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:28.792759 (+ 749us) server_negotiation.cc:300] Negotiation successful
0203 21:07:28.792904 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260203 21:07:28.793058 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.779791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:56231 (local address 127.0.102.188:45925)
0203 21:07:28.779911 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:28.779915 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:28.780027 (+ 112us) server_negotiation.cc:408] Connection header received
0203 21:07:28.781700 (+ 1673us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:28.781704 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:28.781758 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:28.781807 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:28.783080 (+ 1273us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.783865 (+ 785us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.784822 (+ 957us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.785021 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.789037 (+ 4016us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:28.789059 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:28.789062 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:28.789093 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:28.791052 (+ 1959us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:28.791476 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:28.791479 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:28.791481 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:28.791528 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:28.791767 (+ 239us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:28.791770 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:28.791772 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:28.791942 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:28.792010 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:28.792759 (+ 749us) server_negotiation.cc:300] Negotiation successful
0203 21:07:28.792889 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260203 21:07:28.793359 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.778544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:51099 (local address 127.0.102.189:38693)
0203 21:07:28.778763 (+ 219us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:28.778768 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:28.779365 (+ 597us) server_negotiation.cc:408] Connection header received
0203 21:07:28.781102 (+ 1737us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:28.781107 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:28.781177 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:28.781229 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:28.783080 (+ 1851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.783866 (+ 786us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.784822 (+ 956us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.785021 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.789006 (+ 3985us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:28.789031 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:28.789034 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:28.789072 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:28.791611 (+ 2539us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:28.792081 (+ 470us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:28.792085 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:28.792087 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:28.792145 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:28.792416 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:28.792419 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:28.792422 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:28.792615 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:28.792715 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:28.793077 (+ 362us) server_negotiation.cc:300] Negotiation successful
0203 21:07:28.793207 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260203 21:07:28.794646 8157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.779722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.188:45925 (local address 127.0.102.129:56231)
0203 21:07:28.779961 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:28.779972 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:28.781571 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:28.781944 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:28.781947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:28.782120 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:28.782910 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.782928 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.784019 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.784022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:28.784690 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.784699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.785260 (+ 561us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.786054 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:28.786073 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:28.788748 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:28.791188 (+ 2440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:28.791191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:28.791193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:28.791370 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:28.791624 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:28.791627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:28.791629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:28.791679 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:28.792098 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:28.792101 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:28.792425 (+ 324us) client_negotiation.cc:769] Sending connection context
0203 21:07:28.793296 (+ 871us) client_negotiation.cc:240] Negotiation successful
0203 21:07:28.793695 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"spinlock_wait_cycles":41088,"thread_start_us":79,"threads_started":1}
I20260203 21:07:28.794651 8156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.778771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.189:38693 (local address 127.0.102.129:51099)
0203 21:07:28.779233 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:28.779274 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:28.780697 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:28.781522 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:28.781533 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:28.782104 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:28.782910 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.782928 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.784019 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.784022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:28.784690 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.784699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.785260 (+ 561us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.786014 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:28.786054 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:28.788845 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:28.791743 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:28.791746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:28.791748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:28.791960 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:28.792262 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:28.792265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:28.792268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:28.792321 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:28.792787 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:28.792790 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:28.792881 (+ 91us) client_negotiation.cc:769] Sending connection context
0203 21:07:28.793508 (+ 627us) client_negotiation.cc:240] Negotiation successful
0203 21:07:28.793979 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":358,"spinlock_wait_cycles":29568,"thread_start_us":99,"threads_started":1}
I20260203 21:07:28.794759 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.778771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.129:37471)
0203 21:07:28.779235 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:28.779274 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:28.780871 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:28.781807 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:28.781811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:28.782620 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:28.782942 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.782950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.784021 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.784025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:28.784693 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.784702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.785260 (+ 558us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.786166 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:28.786183 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:28.788664 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:28.790898 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:28.790905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:28.790924 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:28.791243 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:28.791525 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:28.791528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:28.791530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:28.791646 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:28.792090 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:28.792097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:28.792425 (+ 328us) client_negotiation.cc:769] Sending connection context
0203 21:07:28.793745 (+ 1320us) client_negotiation.cc:240] Negotiation successful
0203 21:07:28.793863 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"spinlock_wait_cycles":10752,"thread_start_us":105,"threads_started":1}
I20260203 21:07:28.796090 8151 heartbeater.cc:344] Connected to a master server at 127.0.102.188:45925
I20260203 21:07:28.796365 8151 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:28.796514 8152 heartbeater.cc:344] Connected to a master server at 127.0.102.190:42297
I20260203 21:07:28.796589 8152 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:28.796708 8153 heartbeater.cc:344] Connected to a master server at 127.0.102.189:38693
I20260203 21:07:28.796766 8153 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:28.797052 8151 heartbeater.cc:507] Master 127.0.102.188:45925 requested a full tablet report, sending...
I20260203 21:07:28.797045 8153 heartbeater.cc:507] Master 127.0.102.189:38693 requested a full tablet report, sending...
I20260203 21:07:28.797045 8152 heartbeater.cc:507] Master 127.0.102.190:42297 requested a full tablet report, sending...
I20260203 21:07:28.798480 7773 ts_manager.cc:194] Registered new tserver with Master: f1f9e3fc5a4e44edaca392a6e083f178 (127.0.102.129:36543)
I20260203 21:07:28.798596 7956 ts_manager.cc:194] Registered new tserver with Master: f1f9e3fc5a4e44edaca392a6e083f178 (127.0.102.129:36543)
I20260203 21:07:28.799280 7161 ts_manager.cc:194] Re-registered known tserver with Master: f1f9e3fc5a4e44edaca392a6e083f178 (127.0.102.129:36543)
I20260203 21:07:28.799965 7161 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:37471
W20260203 21:07:28.896701 8158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:28.896931 8158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:28.896993 8158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:28.900225 8158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:28.900286 8158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:28.900364 8158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:07:28.904493 8158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:45287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=37805
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:42297,127.0.102.189:38693,127.0.102.188:45925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:28.905594 8158 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:28.906435 8158 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:28.912974 8164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:28.912974 8163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:28.913030 8166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:28.913112 8158 server_base.cc:1060] running on GCE node
I20260203 21:07:28.913642 8158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:28.914214 8158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:28.915377 8158 hybrid_clock.cc:648] HybridClock initialized: now 1770152848915360 us; error 31 us; skew 500 ppm
Feb 03 21:07:28 dist-test-slave-j889 krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:28.918006 8158 init.cc:377] Logged in from keytab as oryx/127.0.102.130@KRBTEST.COM (short username oryx)
I20260203 21:07:28.919059 8158 webserver.cc:492] Webserver started at http://127.0.102.130:37805/ using document root <none> and password file <none>
I20260203 21:07:28.919588 8158 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:28.919651 8158 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:28.923051 8158 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:28.924829 8173 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:28.925751 8158 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:28.925846 8158 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f"
format_stamp: "Formatted at 2026-02-03 21:07:25 on dist-test-slave-j889"
server_key: "fd0f3ac7a6f709107d786045f5b14964"
server_key_iv: "8999dc38d823f8326df9ef149e07494d"
server_key_version: "encryptionkey@0"
I20260203 21:07:28.926241 8158 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:28.942377 8158 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:28.945122 8158 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:28.945295 8158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:28.945894 8158 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:28.946811 8158 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:28.946866 8158 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:28.946918 8158 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:28.946941 8158 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:28.955204 8158 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:45287
I20260203 21:07:28.955228 8286 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:45287 every 8 connection(s)
I20260203 21:07:28.956202 8158 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:07:28.963379 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 8158
I20260203 21:07:28.963553 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 7498
Feb 03 21:07:28 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM
Feb 03 21:07:28 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.188@KRBTEST.COM
Feb 03 21:07:28 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.189@KRBTEST.COM
I20260203 21:07:28.970860 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.959357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:43011 (local address 127.0.102.188:45925)
0203 21:07:28.959477 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:28.959481 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:28.959608 (+ 127us) server_negotiation.cc:408] Connection header received
0203 21:07:28.960014 (+ 406us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:28.960018 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:28.960071 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:28.960142 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:28.961465 (+ 1323us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.962229 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.962847 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.963008 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.966245 (+ 3237us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:28.966265 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:28.966268 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:28.966297 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:28.968614 (+ 2317us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:28.969393 (+ 779us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:28.969396 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:28.969398 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:28.969440 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:28.969673 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:28.969677 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:28.969679 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:28.969858 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:28.969964 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:28.970575 (+ 611us) server_negotiation.cc:300] Negotiation successful
0203 21:07:28.970696 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260203 21:07:28.970860 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.957825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:39027 (local address 127.0.102.190:42297)
0203 21:07:28.957967 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:28.957970 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:28.958903 (+ 933us) server_negotiation.cc:408] Connection header received
0203 21:07:28.960042 (+ 1139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:28.960046 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:28.960102 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:28.960150 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:28.961548 (+ 1398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.962047 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.962788 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.962982 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.966023 (+ 3041us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:28.966044 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:28.966048 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:28.966078 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:28.967576 (+ 1498us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:28.969180 (+ 1604us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:28.969184 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:28.969186 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:28.969235 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:28.969754 (+ 519us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:28.969757 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:28.969758 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:28.969906 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:28.969993 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:28.970572 (+ 579us) server_negotiation.cc:300] Negotiation successful
0203 21:07:28.970696 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260203 21:07:28.971529 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.958043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:44585 (local address 127.0.102.189:38693)
0203 21:07:28.958153 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:28.958156 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:28.958877 (+ 721us) server_negotiation.cc:408] Connection header received
0203 21:07:28.960042 (+ 1165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:28.960046 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:28.960099 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:28.960164 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:28.961502 (+ 1338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.962228 (+ 726us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.962943 (+ 715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.963129 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.967034 (+ 3905us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:28.967055 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:28.967058 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:28.967088 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:28.968523 (+ 1435us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:28.969368 (+ 845us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:28.969371 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:28.969373 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:28.969439 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:28.970197 (+ 758us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:28.970200 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:28.970202 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:28.970357 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:28.970436 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:28.971270 (+ 834us) server_negotiation.cc:300] Negotiation successful
0203 21:07:28.971378 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260203 21:07:28.971693 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.958255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.130:39027)
0203 21:07:28.958739 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:28.958784 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:28.959827 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:28.960296 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:28.960300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:28.961181 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:28.961424 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.961431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.962162 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.962166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:28.962677 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.962686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.962873 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.963462 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:28.963497 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:28.965820 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:28.968767 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:28.968770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:28.968772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:28.969085 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:28.969321 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:28.969324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:28.969326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:28.969428 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:28.970008 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:28.970011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:28.970290 (+ 279us) client_negotiation.cc:769] Sending connection context
0203 21:07:28.970526 (+ 236us) client_negotiation.cc:240] Negotiation successful
0203 21:07:28.970790 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":25,"spinlock_wait_cycles":55680,"thread_start_us":109,"threads_started":1}
I20260203 21:07:28.971935 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.958255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.189:38693 (local address 127.0.102.130:44585)
0203 21:07:28.958739 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:28.958784 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:28.959827 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:28.960296 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:28.960299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:28.960772 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:28.961351 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.961359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.962323 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.962326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:28.962826 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.962833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.963247 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.964389 (+ 1142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:28.964405 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:28.966885 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:28.968660 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:28.968666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:28.968678 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:28.969078 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:28.970051 (+ 973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:28.970054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:28.970055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:28.970103 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:28.970483 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:28.970995 (+ 512us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:28.971081 (+ 86us) client_negotiation.cc:769] Sending connection context
0203 21:07:28.971643 (+ 562us) client_negotiation.cc:240] Negotiation successful
0203 21:07:28.971767 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"spinlock_wait_cycles":36480,"thread_start_us":132,"threads_started":1}
I20260203 21:07:28.971937 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:34113
--local_ip_for_outbound_sockets=127.0.102.131
--tserver_master_addrs=127.0.102.190:42297
--webserver_port=40259
--webserver_interface=127.0.102.131
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.102.190:42297,127.0.102.189:38693
--tserver_master_addrs=127.0.102.190:42297,127.0.102.189:38693,127.0.102.188:45925 with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260203 21:07:28.973057 8289 heartbeater.cc:344] Connected to a master server at 127.0.102.189:38693
I20260203 21:07:28.973212 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:28.959260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.188:45925 (local address 127.0.102.130:43011)
0203 21:07:28.959529 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:28.959559 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:28.959834 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:28.960288 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:28.960296 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:28.960758 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:28.961331 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.961342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.962373 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:28.962376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:28.962732 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:28.962739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:28.963098 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:28.964161 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:28.964177 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:28.966124 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:28.969106 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:28.969109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:28.969111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:28.969294 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:28.969541 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:28.969543 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:28.969545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:28.969588 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:28.969983 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:28.969989 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:28.970291 (+ 302us) client_negotiation.cc:769] Sending connection context
0203 21:07:28.971859 (+ 1568us) client_negotiation.cc:240] Negotiation successful
0203 21:07:28.971967 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"spinlock_wait_cycles":14976,"thread_start_us":73,"threads_started":1}
I20260203 21:07:28.973240 8288 heartbeater.cc:344] Connected to a master server at 127.0.102.190:42297
I20260203 21:07:28.973300 8289 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:28.973309 8288 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:28.973716 8287 heartbeater.cc:344] Connected to a master server at 127.0.102.188:45925
I20260203 21:07:28.973790 8287 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:28.973896 8288 heartbeater.cc:507] Master 127.0.102.190:42297 requested a full tablet report, sending...
I20260203 21:07:28.973896 8289 heartbeater.cc:507] Master 127.0.102.189:38693 requested a full tablet report, sending...
I20260203 21:07:28.973963 8287 heartbeater.cc:507] Master 127.0.102.188:45925 requested a full tablet report, sending...
I20260203 21:07:28.975126 7773 ts_manager.cc:194] Registered new tserver with Master: 2e5f9dfb6e7a4fc0bb51f68658d4b05f (127.0.102.130:45287)
I20260203 21:07:28.975134 7956 ts_manager.cc:194] Registered new tserver with Master: 2e5f9dfb6e7a4fc0bb51f68658d4b05f (127.0.102.130:45287)
I20260203 21:07:28.975606 7161 ts_manager.cc:194] Re-registered known tserver with Master: 2e5f9dfb6e7a4fc0bb51f68658d4b05f (127.0.102.130:45287)
I20260203 21:07:28.976135 7161 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.102.130@KRBTEST.COM'} at 127.0.102.130:39027
W20260203 21:07:29.077869 8294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:29.078099 8294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:29.078178 8294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:29.081542 8294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:29.081610 8294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:29.081707 8294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:07:29.086009 8294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:34113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=40259
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:42297,127.0.102.189:38693,127.0.102.188:45925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:29.087046 8294 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:29.087852 8294 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:29.094384 8300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:29.094386 8302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:29.094383 8299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:29.094616 8294 server_base.cc:1060] running on GCE node
I20260203 21:07:29.094982 8294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:29.095475 8294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:29.096658 8294 hybrid_clock.cc:648] HybridClock initialized: now 1770152849096634 us; error 37 us; skew 500 ppm
Feb 03 21:07:29 dist-test-slave-j889 krb5kdc[7114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152849, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:29.099264 8294 init.cc:377] Logged in from keytab as oryx/127.0.102.131@KRBTEST.COM (short username oryx)
I20260203 21:07:29.100211 8294 webserver.cc:492] Webserver started at http://127.0.102.131:40259/ using document root <none> and password file <none>
I20260203 21:07:29.100750 8294 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:29.100824 8294 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:29.104107 8294 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:07:29.106048 8309 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:29.107127 8294 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260203 21:07:29.107235 8294 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "a8644652b1a349dbbb003fe66b88e94f"
format_stamp: "Formatted at 2026-02-03 21:07:25 on dist-test-slave-j889"
server_key: "33bcee75ca232c36e5f843740a47ea11"
server_key_iv: "7a389bdc37e446af688fdccda5970ea1"
server_key_version: "encryptionkey@0"
I20260203 21:07:29.107671 8294 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:29.121567 8294 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:29.124277 8294 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:29.124444 8294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:29.125011 8294 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:29.125857 8294 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:29.125908 8294 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:29.125949 8294 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:29.125977 8294 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:29.134737 8294 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:34113
I20260203 21:07:29.134840 8422 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:34113 every 8 connection(s)
I20260203 21:07:29.135700 8294 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
I20260203 21:07:29.138401 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 8294
Feb 03 21:07:29 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152849, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.189@KRBTEST.COM
Feb 03 21:07:29 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152849, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM
Feb 03 21:07:29 dist-test-slave-j889 krb5kdc[7114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152849, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.188@KRBTEST.COM
I20260203 21:07:29.150419 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.137391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:51039 (local address 127.0.102.189:38693)
0203 21:07:29.137507 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:29.137510 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:29.138148 (+ 638us) server_negotiation.cc:408] Connection header received
0203 21:07:29.139157 (+ 1009us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:29.139161 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:29.139218 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:29.139271 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:29.141287 (+ 2016us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.141982 (+ 695us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.142889 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.143066 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.146425 (+ 3359us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:29.146443 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:29.146446 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:29.146476 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:29.148265 (+ 1789us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:29.148669 (+ 404us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:29.148673 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:29.148675 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:29.148737 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:29.149068 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:29.149072 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:29.149075 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:29.149268 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:29.149363 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:29.150129 (+ 766us) server_negotiation.cc:300] Negotiation successful
0203 21:07:29.150251 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260203 21:07:29.151276 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.137641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.189:38693 (local address 127.0.102.131:51039)
0203 21:07:29.138023 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:29.138058 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:29.138927 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:29.139557 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:29.139568 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:29.140576 (+ 1008us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:29.141138 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:29.141163 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.142138 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.142142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:29.142781 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:29.142790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.143278 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.143916 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:29.143951 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:29.145732 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:29.148376 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:29.148379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:29.148381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:29.148553 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:29.148838 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:29.148841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:29.148843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:29.148978 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:29.149511 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:29.149518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:29.149841 (+ 323us) client_negotiation.cc:769] Sending connection context
0203 21:07:29.150095 (+ 254us) client_negotiation.cc:240] Negotiation successful
0203 21:07:29.150390 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"spinlock_wait_cycles":3968,"thread_start_us":93,"threads_started":1}
I20260203 21:07:29.152217 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.139169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:50957 (local address 127.0.102.190:42297)
0203 21:07:29.139315 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:29.139319 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:29.139331 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:29.139369 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:29.139372 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:29.139414 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:29.139475 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:29.141274 (+ 1799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.142042 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.142872 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.143064 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.146218 (+ 3154us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:29.146239 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:29.146243 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:29.146271 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:29.147807 (+ 1536us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:29.148703 (+ 896us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:29.148706 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:29.148709 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:29.148762 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:29.149216 (+ 454us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:29.149219 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:29.149222 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:29.149383 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:29.149455 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:29.151959 (+ 2504us) server_negotiation.cc:300] Negotiation successful
0203 21:07:29.152081 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260203 21:07:29.152685 8425 heartbeater.cc:344] Connected to a master server at 127.0.102.189:38693
I20260203 21:07:29.152942 8425 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:29.153741 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.138513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.131:50957)
0203 21:07:29.138788 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:29.138801 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:29.138927 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:29.139562 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:29.139568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:29.140176 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:29.141136 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:29.141150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.142175 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.142179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:29.142758 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:29.142766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.143277 (+ 511us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.144669 (+ 1392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:29.144683 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:29.146080 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:29.147936 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:29.147943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:29.147957 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:29.148281 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:29.149087 (+ 806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:29.149090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:29.149092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:29.149131 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:29.151078 (+ 1947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:29.151082 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:29.151172 (+ 90us) client_negotiation.cc:769] Sending connection context
0203 21:07:29.151801 (+ 629us) client_negotiation.cc:240] Negotiation successful
0203 21:07:29.152979 (+ 1178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"spinlock_wait_cycles":69248,"thread_start_us":106,"threads_started":1}
I20260203 21:07:29.153791 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.140153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.0.102.189:38693)
0203 21:07:29.140411 (+ 258us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:29.140415 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:29.141215 (+ 800us) server_negotiation.cc:408] Connection header received
0203 21:07:29.141715 (+ 500us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:29.141718 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:29.141764 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:29.141817 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:29.142881 (+ 1064us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.143545 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.144388 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.144565 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.145937 (+ 1372us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:29.145960 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:29.145964 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:29.145990 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:29.150957 (+ 4967us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:29.152737 (+ 1780us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:29.152741 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:29.152743 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:29.152789 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:29.153031 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:29.153035 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:29.153037 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:29.153182 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:29.153275 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:29.153534 (+ 259us) server_negotiation.cc:300] Negotiation successful
0203 21:07:29.153656 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
I20260203 21:07:29.153868 8425 heartbeater.cc:507] Master 127.0.102.189:38693 requested a full tablet report, sending...
I20260203 21:07:29.154675 8424 heartbeater.cc:344] Connected to a master server at 127.0.102.190:42297
I20260203 21:07:29.154752 8424 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:29.154882 7773 ts_manager.cc:194] Registered new tserver with Master: a8644652b1a349dbbb003fe66b88e94f (127.0.102.131:34113)
I20260203 21:07:29.154933 8424 heartbeater.cc:507] Master 127.0.102.190:42297 requested a full tablet report, sending...
I20260203 21:07:29.155121 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.139382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:46079 (local address 127.0.102.188:45925)
0203 21:07:29.139496 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:29.139500 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:29.141063 (+ 1563us) server_negotiation.cc:408] Connection header received
0203 21:07:29.141244 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:29.141248 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:29.141298 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:29.141347 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:29.142030 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.142746 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.143506 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.144093 (+ 587us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.148193 (+ 4100us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:29.148215 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:29.148218 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:29.148250 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:29.149945 (+ 1695us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:29.151733 (+ 1788us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:29.151737 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:29.151739 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:29.151789 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:29.152053 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:29.152266 (+ 213us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:29.152269 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:29.152620 (+ 351us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:29.153847 (+ 1227us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:29.154857 (+ 1010us) server_negotiation.cc:300] Negotiation successful
0203 21:07:29.154976 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260203 21:07:29.155524 7161 ts_manager.cc:194] Re-registered known tserver with Master: a8644652b1a349dbbb003fe66b88e94f (127.0.102.131:34113)
I20260203 21:07:29.155575 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.138922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.188:45925 (local address 127.0.102.131:46079)
0203 21:07:29.140990 (+ 2068us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:29.141003 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:29.141121 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:29.141463 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:29.141466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:29.141630 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:29.141905 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:29.141912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.142871 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.142874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:29.143378 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:29.143386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.144214 (+ 828us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.144661 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:29.144677 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:29.147556 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:29.151116 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:29.151119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:29.151121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:29.151532 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:29.151864 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:29.151867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:29.151870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:29.151952 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:29.154574 (+ 2622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:29.154577 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:29.154664 (+ 87us) client_negotiation.cc:769] Sending connection context
0203 21:07:29.155284 (+ 620us) client_negotiation.cc:240] Negotiation successful
0203 21:07:29.155410 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2001,"thread_start_us":77,"threads_started":1}
I20260203 21:07:29.156104 8423 heartbeater.cc:344] Connected to a master server at 127.0.102.188:45925
I20260203 21:07:29.156172 8423 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:29.156195 7161 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.102.131@KRBTEST.COM'} at 127.0.102.131:50957
I20260203 21:07:29.156342 8423 heartbeater.cc:507] Master 127.0.102.188:45925 requested a full tablet report, sending...
I20260203 21:07:29.157106 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.140578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.0.102.190:42297)
0203 21:07:29.144183 (+ 3605us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:29.144188 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:29.144200 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:29.144231 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:29.144234 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:29.144276 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:29.147527 (+ 3251us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:29.147640 (+ 113us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.148378 (+ 738us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.150786 (+ 2408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.150958 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.152114 (+ 1156us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:29.152131 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:29.152135 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:29.152169 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:29.155343 (+ 3174us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:29.155754 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:29.155757 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:29.155759 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:29.155806 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:29.156044 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:29.156047 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:29.156049 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:29.156225 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:29.156341 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:29.156863 (+ 522us) server_negotiation.cc:300] Negotiation successful
0203 21:07:29.156973 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3518,"thread_start_us":87,"threads_started":1}
I20260203 21:07:29.157495 7956 ts_manager.cc:194] Registered new tserver with Master: a8644652b1a349dbbb003fe66b88e94f (127.0.102.131:34113)
I20260203 21:07:29.159050 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.140993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.0.102.188:45925)
0203 21:07:29.145551 (+ 4558us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:29.145556 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:29.145568 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:29.145600 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:29.145603 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:29.145649 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:29.145733 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:29.148513 (+ 2780us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.149253 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.150217 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.150390 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.151837 (+ 1447us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:29.151855 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:29.151858 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:29.151880 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:29.157020 (+ 5140us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:29.157757 (+ 737us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:29.157761 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:29.157763 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:29.157805 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:29.158053 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:29.158057 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:29.158059 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:29.158235 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:29.158325 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:29.158610 (+ 285us) server_negotiation.cc:300] Negotiation successful
0203 21:07:29.158904 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4478,"thread_start_us":62,"threads_started":1}
I20260203 21:07:29.160029 7161 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260203 21:07:29.162249 7161 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:07:29.178015 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.174224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:45287 (local address 127.0.0.1:32864)
0203 21:07:29.174773 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:29.174785 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:29.174850 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:29.175409 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:29.175412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:29.175428 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:29.175640 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:29.175646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.176751 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.176754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:29.177638 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:29.177647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.177750 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.177769 (+ 19us) client_negotiation.cc:769] Sending connection context
0203 21:07:29.177811 (+ 42us) client_negotiation.cc:240] Negotiation successful
0203 21:07:29.177872 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":94,"threads_started":1}
I20260203 21:07:29.178529 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.174340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.0.102.130:45287)
0203 21:07:29.175040 (+ 700us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:29.175046 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:29.175065 (+ 19us) server_negotiation.cc:408] Connection header received
0203 21:07:29.175134 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:29.175138 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:29.175282 (+ 144us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:29.175394 (+ 112us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:29.175751 (+ 357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.176550 (+ 799us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.177773 (+ 1223us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.178189 (+ 416us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.178305 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:29.178365 (+ 60us) server_negotiation.cc:300] Negotiation successful
0203 21:07:29.178426 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":101,"threads_started":1}
I20260203 21:07:29.178709 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.173949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:36543 (local address 127.0.0.1:51514)
0203 21:07:29.174509 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:29.174524 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:29.174605 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:29.175748 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:29.175750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:29.175769 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:29.176005 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:29.176011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.177649 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.177652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:29.178384 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:29.178392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.178482 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.178500 (+ 18us) client_negotiation.cc:769] Sending connection context
0203 21:07:29.178544 (+ 44us) client_negotiation.cc:240] Negotiation successful
0203 21:07:29.178594 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":87,"threads_started":1}
I20260203 21:07:29.179596 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.174173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51514 (local address 127.0.102.129:36543)
0203 21:07:29.175056 (+ 883us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:29.175061 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:29.175079 (+ 18us) server_negotiation.cc:408] Connection header received
0203 21:07:29.175143 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:29.175147 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:29.175282 (+ 135us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:29.175397 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:29.176453 (+ 1056us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.177538 (+ 1085us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.178494 (+ 956us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.179030 (+ 536us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.179114 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:29.179174 (+ 60us) server_negotiation.cc:300] Negotiation successful
0203 21:07:29.179233 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":94,"threads_started":1}
I20260203 21:07:29.179853 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.174240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:34113 (local address 127.0.0.1:58294)
0203 21:07:29.175552 (+ 1312us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:29.175563 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:29.175616 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:29.176107 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:29.176110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:29.176126 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:29.176326 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:29.176331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.178696 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.178700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:29.179522 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:29.179530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.179628 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.179644 (+ 16us) client_negotiation.cc:769] Sending connection context
0203 21:07:29.179688 (+ 44us) client_negotiation.cc:240] Negotiation successful
0203 21:07:29.179739 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1253,"thread_start_us":77,"threads_started":1}
I20260203 21:07:29.180603 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.174583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.0.102.131:34113)
0203 21:07:29.175800 (+ 1217us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:29.175804 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:29.175820 (+ 16us) server_negotiation.cc:408] Connection header received
0203 21:07:29.175877 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:29.175880 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:29.175996 (+ 116us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:29.176101 (+ 105us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:29.176427 (+ 326us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.177689 (+ 1262us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.179634 (+ 1945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.180248 (+ 614us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.180343 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:29.180401 (+ 58us) server_negotiation.cc:300] Negotiation successful
0203 21:07:29.180478 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135,"thread_start_us":88,"threads_started":1}
I20260203 21:07:29.181370 8221 tablet_service.cc:1505] Processing CreateTablet for tablet 20d5a888b97f470b95a2e205e18c1e52 (DEFAULT_TABLE table=test-table [id=d370240bb23940c384af9f146075c97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:29.182211 8221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20d5a888b97f470b95a2e205e18c1e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:29.182045 8085 tablet_service.cc:1505] Processing CreateTablet for tablet 20d5a888b97f470b95a2e205e18c1e52 (DEFAULT_TABLE table=test-table [id=d370240bb23940c384af9f146075c97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:29.182811 8085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20d5a888b97f470b95a2e205e18c1e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:29.188112 8357 tablet_service.cc:1505] Processing CreateTablet for tablet 20d5a888b97f470b95a2e205e18c1e52 (DEFAULT_TABLE table=test-table [id=d370240bb23940c384af9f146075c97a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:29.188619 8457 tablet_bootstrap.cc:492] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178: Bootstrap starting.
I20260203 21:07:29.189853 8357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20d5a888b97f470b95a2e205e18c1e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:29.190683 8457 tablet_bootstrap.cc:654] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:29.191366 8457 log.cc:826] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:29.192971 8457 tablet_bootstrap.cc:492] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178: No bootstrap required, opened a new log
I20260203 21:07:29.193014 8458 tablet_bootstrap.cc:492] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Bootstrap starting.
I20260203 21:07:29.193296 8457 ts_tablet_manager.cc:1403] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260203 21:07:29.195331 8458 tablet_bootstrap.cc:654] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:29.196136 8458 log.cc:826] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:29.196242 8457 raft_consensus.cc:359] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } } peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } }
I20260203 21:07:29.196554 8457 raft_consensus.cc:385] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:29.196623 8457 raft_consensus.cc:740] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f9e3fc5a4e44edaca392a6e083f178, State: Initialized, Role: FOLLOWER
I20260203 21:07:29.196656 8460 tablet_bootstrap.cc:492] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f: Bootstrap starting.
I20260203 21:07:29.197368 8457 consensus_queue.cc:260] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } } peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } }
I20260203 21:07:29.198271 8457 ts_tablet_manager.cc:1434] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260203 21:07:29.198581 8458 tablet_bootstrap.cc:492] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: No bootstrap required, opened a new log
I20260203 21:07:29.198783 8458 ts_tablet_manager.cc:1403] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260203 21:07:29.198853 8460 tablet_bootstrap.cc:654] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:29.199679 8152 heartbeater.cc:499] Master 127.0.102.190:42297 was elected leader, sending a full tablet report...
I20260203 21:07:29.199656 8460 log.cc:826] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:29.202793 8458 raft_consensus.cc:359] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } } peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } }
I20260203 21:07:29.202971 8460 tablet_bootstrap.cc:492] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f: No bootstrap required, opened a new log
I20260203 21:07:29.203017 8458 raft_consensus.cc:385] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:29.203075 8458 raft_consensus.cc:740] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5f9dfb6e7a4fc0bb51f68658d4b05f, State: Initialized, Role: FOLLOWER
I20260203 21:07:29.203148 8460 ts_tablet_manager.cc:1403] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260203 21:07:29.203516 8458 consensus_queue.cc:260] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } } peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } }
I20260203 21:07:29.204407 8458 ts_tablet_manager.cc:1434] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260203 21:07:29.204536 8288 heartbeater.cc:499] Master 127.0.102.190:42297 was elected leader, sending a full tablet report...
W20260203 21:07:29.207710 8290 tablet.cc:2378] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260203 21:07:29.208191 8460 raft_consensus.cc:359] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } } peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } }
I20260203 21:07:29.208386 8460 raft_consensus.cc:385] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:29.208455 8460 raft_consensus.cc:740] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8644652b1a349dbbb003fe66b88e94f, State: Initialized, Role: FOLLOWER
I20260203 21:07:29.208863 8460 consensus_queue.cc:260] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } } peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } }
I20260203 21:07:29.209501 8424 heartbeater.cc:499] Master 127.0.102.190:42297 was elected leader, sending a full tablet report...
I20260203 21:07:29.209625 8460 ts_tablet_manager.cc:1434] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260203 21:07:29.278208 8154 tablet.cc:2378] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260203 21:07:29.336727 8462 raft_consensus.cc:493] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:07:29.336895 8462 raft_consensus.cc:515] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } } peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } }
I20260203 21:07:29.338016 8462 leader_election.cc:290] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e5f9dfb6e7a4fc0bb51f68658d4b05f (127.0.102.130:45287), a8644652b1a349dbbb003fe66b88e94f (127.0.102.131:34113)
I20260203 21:07:29.341205 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.338273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:34113 (local address 127.0.102.129:33641)
0203 21:07:29.338605 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:29.338617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:29.338718 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:29.338976 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:29.338979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:29.338996 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:29.339223 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:29.339229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.340140 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.340143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:29.340741 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:29.340748 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.340915 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.340938 (+ 23us) client_negotiation.cc:769] Sending connection context
0203 21:07:29.340991 (+ 53us) client_negotiation.cc:240] Negotiation successful
0203 21:07:29.341047 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":80,"threads_started":1}
I20260203 21:07:29.341266 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.338273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:45287 (local address 127.0.102.129:48001)
0203 21:07:29.338635 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:29.338645 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:29.338766 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:29.339084 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:29.339089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:29.339109 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:29.339335 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:29.339343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.340095 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.340099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:29.340744 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:29.340754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.340948 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.340971 (+ 23us) client_negotiation.cc:769] Sending connection context
0203 21:07:29.341030 (+ 59us) client_negotiation.cc:240] Negotiation successful
0203 21:07:29.341108 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":45,"thread_start_us":106,"threads_started":1}
I20260203 21:07:29.341634 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.338368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:33641 (local address 127.0.102.131:34113)
0203 21:07:29.338526 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:29.338530 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:29.338689 (+ 159us) server_negotiation.cc:408] Connection header received
0203 21:07:29.338838 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:29.338841 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:29.338890 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:29.338950 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:29.339332 (+ 382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.340014 (+ 682us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.340907 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.341373 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.341409 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:29.341467 (+ 58us) server_negotiation.cc:300] Negotiation successful
0203 21:07:29.341522 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260203 21:07:29.341727 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.338384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:48001 (local address 127.0.102.130:45287)
0203 21:07:29.338532 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:29.338536 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:29.338724 (+ 188us) server_negotiation.cc:408] Connection header received
0203 21:07:29.338859 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:29.338862 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:29.338930 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:29.339009 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:29.339451 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.339994 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.340942 (+ 948us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.341435 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.341468 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:29.341519 (+ 51us) server_negotiation.cc:300] Negotiation successful
0203 21:07:29.341604 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260203 21:07:29.342226 8377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20d5a888b97f470b95a2e205e18c1e52" candidate_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8644652b1a349dbbb003fe66b88e94f" is_pre_election: true
I20260203 21:07:29.342298 8241 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20d5a888b97f470b95a2e205e18c1e52" candidate_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" is_pre_election: true
I20260203 21:07:29.342494 8377 raft_consensus.cc:2468] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1f9e3fc5a4e44edaca392a6e083f178 in term 0.
I20260203 21:07:29.342525 8241 raft_consensus.cc:2468] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1f9e3fc5a4e44edaca392a6e083f178 in term 0.
I20260203 21:07:29.342931 8040 leader_election.cc:304] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e5f9dfb6e7a4fc0bb51f68658d4b05f, a8644652b1a349dbbb003fe66b88e94f, f1f9e3fc5a4e44edaca392a6e083f178; no voters:
I20260203 21:07:29.343163 8462 raft_consensus.cc:2804] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:07:29.343235 8462 raft_consensus.cc:493] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:07:29.343281 8462 raft_consensus.cc:3060] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:29.344159 8462 raft_consensus.cc:515] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } } peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } }
I20260203 21:07:29.344476 8462 leader_election.cc:290] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [CANDIDATE]: Term 1 election: Requested vote from peers 2e5f9dfb6e7a4fc0bb51f68658d4b05f (127.0.102.130:45287), a8644652b1a349dbbb003fe66b88e94f (127.0.102.131:34113)
I20260203 21:07:29.344859 8241 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20d5a888b97f470b95a2e205e18c1e52" candidate_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f"
I20260203 21:07:29.344859 8377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20d5a888b97f470b95a2e205e18c1e52" candidate_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8644652b1a349dbbb003fe66b88e94f"
I20260203 21:07:29.344983 8241 raft_consensus.cc:3060] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:29.344985 8377 raft_consensus.cc:3060] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:29.346108 8241 raft_consensus.cc:2468] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1f9e3fc5a4e44edaca392a6e083f178 in term 1.
I20260203 21:07:29.346122 8377 raft_consensus.cc:2468] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1f9e3fc5a4e44edaca392a6e083f178 in term 1.
I20260203 21:07:29.346428 8038 leader_election.cc:304] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e5f9dfb6e7a4fc0bb51f68658d4b05f, f1f9e3fc5a4e44edaca392a6e083f178; no voters:
I20260203 21:07:29.346589 8462 raft_consensus.cc:2804] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:29.346767 8462 raft_consensus.cc:697] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [term 1 LEADER]: Becoming Leader. State: Replica: f1f9e3fc5a4e44edaca392a6e083f178, State: Running, Role: LEADER
I20260203 21:07:29.347090 8462 consensus_queue.cc:237] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } } peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } }
I20260203 21:07:29.350450 7161 catalog_manager.cc:5654] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1f9e3fc5a4e44edaca392a6e083f178 (127.0.102.129). New cstate: current_term: 1 leader_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } health_report { overall_health: UNKNOWN } } }
I20260203 21:07:29.374941 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.372022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.0.102.129:36543)
0203 21:07:29.372164 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:29.372169 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:29.372192 (+ 23us) server_negotiation.cc:408] Connection header received
0203 21:07:29.372271 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:29.372274 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:29.372326 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:29.372405 (+ 79us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:07:29.372770 (+ 365us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.373272 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.373905 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.374079 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.374171 (+ 92us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:07:29.374592 (+ 421us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:07:29.374701 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:29.374762 (+ 61us) server_negotiation.cc:300] Negotiation successful
0203 21:07:29.374811 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260203 21:07:29.380220 8241 raft_consensus.cc:1275] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 1 FOLLOWER]: Refusing update from remote peer f1f9e3fc5a4e44edaca392a6e083f178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:29.380220 8377 raft_consensus.cc:1275] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f [term 1 FOLLOWER]: Refusing update from remote peer f1f9e3fc5a4e44edaca392a6e083f178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:29.380883 8468 consensus_queue.cc:1048] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:29.381067 8462 consensus_queue.cc:1048] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260203 21:07:29.387157 8426 tablet.cc:2378] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260203 21:07:29.389540 8473 mvcc.cc:204] Tried to move back new op lower bound from 7250546071054417920 to 7250546070928556032. Current Snapshot: MvccSnapshot[applied={T|T < 7250546071054417920}]
I20260203 21:07:29.397589 7162 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:07:29.397758 7162 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:07:29.403357 7162 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 61f5416a4138441e8bdfa4f27aeb963d: Sending DeleteTablet for 3 replicas of tablet 20d5a888b97f470b95a2e205e18c1e52
I20260203 21:07:29.404011 8221 tablet_service.cc:1552] Processing DeleteTablet for tablet 20d5a888b97f470b95a2e205e18c1e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:29 UTC) from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:32864
I20260203 21:07:29.404197 8085 tablet_service.cc:1552] Processing DeleteTablet for tablet 20d5a888b97f470b95a2e205e18c1e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:29 UTC) from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:51514
I20260203 21:07:29.404270 8357 tablet_service.cc:1552] Processing DeleteTablet for tablet 20d5a888b97f470b95a2e205e18c1e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:29 UTC) from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:58294
I20260203 21:07:29.404462 8356 tablet_service.cc:1552] Processing DeleteTablet for tablet 20d5a888b97f470b95a2e205e18c1e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:29 UTC) from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:58294
I20260203 21:07:29.404785 8480 tablet_replica.cc:333] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: stopping tablet replica
I20260203 21:07:29.405084 8480 raft_consensus.cc:2243] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:29.405488 8480 raft_consensus.cc:2272] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:29.405594 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 7130
I20260203 21:07:29.406425 8481 tablet_replica.cc:333] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f: stopping tablet replica
I20260203 21:07:29.406663 8481 raft_consensus.cc:2243] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:29.406939 8481 raft_consensus.cc:2272] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:29.407298 8482 tablet_replica.cc:333] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178: stopping tablet replica
I20260203 21:07:29.407541 8480 ts_tablet_manager.cc:1916] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:29.407559 8482 raft_consensus.cc:2243] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [term 1 LEADER]: Raft consensus shutting down.
I20260203 21:07:29.407912 8482 raft_consensus.cc:2272] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:29.408260 8481 ts_tablet_manager.cc:1916] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:29.409191 8482 ts_tablet_manager.cc:1916] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:29.410367 8480 ts_tablet_manager.cc:1929] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:07:29.410504 8480 log.cc:1199] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/wal/wals/20d5a888b97f470b95a2e205e18c1e52
I20260203 21:07:29.410633 8481 ts_tablet_manager.cc:1929] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:07:29.410781 8481 log.cc:1199] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/wal/wals/20d5a888b97f470b95a2e205e18c1e52
I20260203 21:07:29.410877 8480 ts_tablet_manager.cc:1950] T 20d5a888b97f470b95a2e205e18c1e52 P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Deleting consensus metadata
I20260203 21:07:29.411433 8481 ts_tablet_manager.cc:1950] T 20d5a888b97f470b95a2e205e18c1e52 P a8644652b1a349dbbb003fe66b88e94f: Deleting consensus metadata
I20260203 21:07:29.411850 8482 ts_tablet_manager.cc:1929] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:07:29.411985 8482 log.cc:1199] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-0/wal/wals/20d5a888b97f470b95a2e205e18c1e52
I20260203 21:07:29.412388 8482 ts_tablet_manager.cc:1950] T 20d5a888b97f470b95a2e205e18c1e52 P f1f9e3fc5a4e44edaca392a6e083f178: Deleting consensus metadata
W20260203 21:07:29.413352 8311 connection.cc:537] server connection from 127.0.0.1:58294 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260203 21:07:29.413398 8174 connection.cc:537] server connection from 127.0.0.1:32864 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260203 21:07:29.424923 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.416260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.0.102.189:38693)
0203 21:07:29.416390 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:29.416395 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:29.416652 (+ 257us) server_negotiation.cc:408] Connection header received
0203 21:07:29.416804 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:29.416808 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:29.416863 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:29.416963 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:29.418229 (+ 1266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.419047 (+ 818us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.419813 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.420012 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.421748 (+ 1736us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:29.421764 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:29.421767 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:29.421792 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:29.423462 (+ 1670us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:29.423920 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:29.423923 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:29.423925 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:29.423970 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:29.424174 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:29.424177 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:29.424179 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:29.424321 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:29.424441 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:29.424682 (+ 241us) server_negotiation.cc:300] Negotiation successful
0203 21:07:29.424798 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260203 21:07:29.425954 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:29.416719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55228 (local address 127.0.102.188:45925)
0203 21:07:29.416844 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:29.416848 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:29.417251 (+ 403us) server_negotiation.cc:408] Connection header received
0203 21:07:29.417365 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:29.417369 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:29.417422 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:29.417507 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:29.418850 (+ 1343us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.419505 (+ 655us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:29.420561 (+ 1056us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:29.420999 (+ 438us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:29.421676 (+ 677us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:29.421694 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:29.421697 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:29.421728 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:29.423387 (+ 1659us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:29.423855 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:29.423859 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:29.423861 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:29.423919 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:29.424483 (+ 564us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:29.424487 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:29.424489 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:29.424663 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:29.424771 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:29.425508 (+ 737us) server_negotiation.cc:300] Negotiation successful
0203 21:07:29.425793 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260203 21:07:30.394891 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:30.394461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.129:45377)
0203 21:07:30.394712 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:30.394794 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260203 21:07:30.394914 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:30.394461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.130:52293)
0203 21:07:30.394769 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:30.394830 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":83,"threads_started":1}
W20260203 21:07:30.395187 8288 heartbeater.cc:646] Failed to heartbeat to 127.0.102.190:42297 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
W20260203 21:07:30.395246 8152 heartbeater.cc:646] Failed to heartbeat to 127.0.102.190:42297 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
I20260203 21:07:30.405082 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:30.404677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.131:33953)
0203 21:07:30.404906 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:30.404975 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":70,"threads_started":1}
W20260203 21:07:30.405326 8424 heartbeater.cc:646] Failed to heartbeat to 127.0.102.190:42297 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
I20260203 21:07:30.902669 8492 raft_consensus.cc:493] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 61f5416a4138441e8bdfa4f27aeb963d)
I20260203 21:07:30.902868 8492 raft_consensus.cc:515] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } }
I20260203 21:07:30.903030 8493 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 61f5416a4138441e8bdfa4f27aeb963d)
I20260203 21:07:30.903177 8493 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } }
I20260203 21:07:30.903879 8492 leader_election.cc:290] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61f5416a4138441e8bdfa4f27aeb963d (127.0.102.190:42297), 0ef1705ecf404c62b8d1ca281e959679 (127.0.102.188:45925)
I20260203 21:07:30.904002 8493 leader_election.cc:290] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61f5416a4138441e8bdfa4f27aeb963d (127.0.102.190:42297), c31601e9963b484bbb7869e10d137467 (127.0.102.189:38693)
I20260203 21:07:30.904459 7783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ef1705ecf404c62b8d1ca281e959679" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c31601e9963b484bbb7869e10d137467" is_pre_election: true
I20260203 21:07:30.904556 7966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31601e9963b484bbb7869e10d137467" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ef1705ecf404c62b8d1ca281e959679" is_pre_election: true
I20260203 21:07:30.904614 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:30.904090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:59592)
0203 21:07:30.904447 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:30.904516 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":85,"threads_started":1}
I20260203 21:07:30.904654 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:30.904197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:59594)
0203 21:07:30.904447 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:30.904516 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":79,"threads_started":1}
I20260203 21:07:30.904659 7783 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ef1705ecf404c62b8d1ca281e959679 in term 1.
I20260203 21:07:30.904701 7966 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c31601e9963b484bbb7869e10d137467 in term 1.
W20260203 21:07:30.904767 7758 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
W20260203 21:07:30.904891 7941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
I20260203 21:07:30.904983 7943 leader_election.cc:304] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ef1705ecf404c62b8d1ca281e959679, c31601e9963b484bbb7869e10d137467; no voters:
I20260203 21:07:30.905190 8493 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260203 21:07:30.905187 7758 leader_election.cc:304] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ef1705ecf404c62b8d1ca281e959679, c31601e9963b484bbb7869e10d137467; no voters:
I20260203 21:07:30.905254 8493 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 61f5416a4138441e8bdfa4f27aeb963d)
I20260203 21:07:30.905294 8493 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 1 FOLLOWER]: Advancing to term 2
I20260203 21:07:30.905359 8492 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260203 21:07:30.905416 8492 raft_consensus.cc:493] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 61f5416a4138441e8bdfa4f27aeb963d)
I20260203 21:07:30.905457 8492 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 1 FOLLOWER]: Advancing to term 2
I20260203 21:07:30.905715 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:30.905481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:59598)
0203 21:07:30.905613 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:30.905655 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260203 21:07:30.905727 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:30.905454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:59596)
0203 21:07:30.905617 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:30.905660 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":23}
W20260203 21:07:30.905803 7758 leader_election.cc:336] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 61f5416a4138441e8bdfa4f27aeb963d (127.0.102.190:42297): Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
W20260203 21:07:30.905819 7941 leader_election.cc:336] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 61f5416a4138441e8bdfa4f27aeb963d (127.0.102.190:42297): Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
I20260203 21:07:30.906479 8493 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } }
I20260203 21:07:30.906479 8492 raft_consensus.cc:515] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } }
I20260203 21:07:30.906781 8493 leader_election.cc:290] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [CANDIDATE]: Term 2 election: Requested vote from peers 61f5416a4138441e8bdfa4f27aeb963d (127.0.102.190:42297), c31601e9963b484bbb7869e10d137467 (127.0.102.189:38693)
I20260203 21:07:30.906787 8492 leader_election.cc:290] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [CANDIDATE]: Term 2 election: Requested vote from peers 61f5416a4138441e8bdfa4f27aeb963d (127.0.102.190:42297), 0ef1705ecf404c62b8d1ca281e959679 (127.0.102.188:45925)
I20260203 21:07:30.907070 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:30.906875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:59602)
0203 21:07:30.906968 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:30.907011 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260203 21:07:30.907173 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:30.906888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:59608)
0203 21:07:30.907025 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:30.907064 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260203 21:07:30.907230 7966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31601e9963b484bbb7869e10d137467" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ef1705ecf404c62b8d1ca281e959679"
I20260203 21:07:30.907377 7966 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c31601e9963b484bbb7869e10d137467 in current term 2: Already voted for candidate 0ef1705ecf404c62b8d1ca281e959679 in this term.
I20260203 21:07:30.907359 7783 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ef1705ecf404c62b8d1ca281e959679" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c31601e9963b484bbb7869e10d137467"
I20260203 21:07:30.907487 7783 raft_consensus.cc:2393] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ef1705ecf404c62b8d1ca281e959679 in current term 2: Already voted for candidate c31601e9963b484bbb7869e10d137467 in this term.
I20260203 21:07:30.907754 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:30.907522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:59622)
0203 21:07:30.907646 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:30.907690 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260203 21:07:30.907833 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:30.907595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:59634)
0203 21:07:30.907742 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:30.907783 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260203 21:07:30.907866 7941 leader_election.cc:336] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 61f5416a4138441e8bdfa4f27aeb963d (127.0.102.190:42297): Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
I20260203 21:07:30.907927 7941 leader_election.cc:304] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ef1705ecf404c62b8d1ca281e959679; no voters: 61f5416a4138441e8bdfa4f27aeb963d, c31601e9963b484bbb7869e10d137467
W20260203 21:07:30.907982 7758 leader_election.cc:336] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 61f5416a4138441e8bdfa4f27aeb963d (127.0.102.190:42297): Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
I20260203 21:07:30.908041 8493 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260203 21:07:30.908052 7758 leader_election.cc:304] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c31601e9963b484bbb7869e10d137467; no voters: 0ef1705ecf404c62b8d1ca281e959679, 61f5416a4138441e8bdfa4f27aeb963d
I20260203 21:07:30.908179 8492 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260203 21:07:31.396157 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:31.395773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.129:45767)
0203 21:07:31.396007 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:31.396063 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260203 21:07:31.396160 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:31.395765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.130:58841)
0203 21:07:31.396008 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:31.396068 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
I20260203 21:07:31.406255 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:31.405826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.131:50519)
0203 21:07:31.406089 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:31.406155 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
I20260203 21:07:32.397250 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.396813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.130:47001)
0203 21:07:32.397089 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.397158 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
I20260203 21:07:32.397250 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.396814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.129:59581)
0203 21:07:32.397112 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.397162 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":80,"threads_started":1}
I20260203 21:07:32.407308 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.406898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.131:47947)
0203 21:07:32.407160 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.407214 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":98,"threads_started":1}
I20260203 21:07:32.459283 8504 raft_consensus.cc:493] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:07:32.459431 8504 raft_consensus.cc:515] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } }
I20260203 21:07:32.459838 8504 leader_election.cc:290] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 61f5416a4138441e8bdfa4f27aeb963d (127.0.102.190:42297), 0ef1705ecf404c62b8d1ca281e959679 (127.0.102.188:45925)
I20260203 21:07:32.460419 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.460041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:59854)
0203 21:07:32.460260 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.460317 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":75,"threads_started":1}
I20260203 21:07:32.460606 7966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31601e9963b484bbb7869e10d137467" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ef1705ecf404c62b8d1ca281e959679" is_pre_election: true
I20260203 21:07:32.460770 7966 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c31601e9963b484bbb7869e10d137467 in term 2.
I20260203 21:07:32.461144 7758 leader_election.cc:304] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ef1705ecf404c62b8d1ca281e959679, c31601e9963b484bbb7869e10d137467; no voters:
I20260203 21:07:32.461216 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.461028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:59862)
0203 21:07:32.461116 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.461159 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260203 21:07:32.461302 7758 leader_election.cc:336] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 61f5416a4138441e8bdfa4f27aeb963d (127.0.102.190:42297): Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
I20260203 21:07:32.461364 8504 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260203 21:07:32.461498 8504 raft_consensus.cc:493] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:07:32.461546 8504 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 2 FOLLOWER]: Advancing to term 3
I20260203 21:07:32.462435 8504 raft_consensus.cc:515] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } }
I20260203 21:07:32.462723 8504 leader_election.cc:290] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [CANDIDATE]: Term 3 election: Requested vote from peers 61f5416a4138441e8bdfa4f27aeb963d (127.0.102.190:42297), 0ef1705ecf404c62b8d1ca281e959679 (127.0.102.188:45925)
I20260203 21:07:32.463075 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.462895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:59876)
0203 21:07:32.462988 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.463028 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260203 21:07:32.463254 7966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c31601e9963b484bbb7869e10d137467" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0ef1705ecf404c62b8d1ca281e959679"
I20260203 21:07:32.463373 7966 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 2 FOLLOWER]: Advancing to term 3
I20260203 21:07:32.463752 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.463528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:59884)
0203 21:07:32.463646 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.463685 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260203 21:07:32.463846 7758 leader_election.cc:336] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 61f5416a4138441e8bdfa4f27aeb963d (127.0.102.190:42297): Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
I20260203 21:07:32.464262 7966 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c31601e9963b484bbb7869e10d137467 in term 3.
I20260203 21:07:32.464538 7758 leader_election.cc:304] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ef1705ecf404c62b8d1ca281e959679, c31601e9963b484bbb7869e10d137467; no voters: 61f5416a4138441e8bdfa4f27aeb963d
I20260203 21:07:32.464721 8504 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 3 FOLLOWER]: Leader election won for term 3
I20260203 21:07:32.464867 8504 raft_consensus.cc:697] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [term 3 LEADER]: Becoming Leader. State: Replica: c31601e9963b484bbb7869e10d137467, State: Running, Role: LEADER
I20260203 21:07:32.465093 8504 consensus_queue.cc:237] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } }
I20260203 21:07:32.466099 8507 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31601e9963b484bbb7869e10d137467. Latest consensus state: current_term: 3 leader_uuid: "c31601e9963b484bbb7869e10d137467" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } } }
I20260203 21:07:32.466182 8507 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:32.466460 8509 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:07:32.469343 8509 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:07:32.469750 8509 catalog_manager.cc:1269] Loaded cluster ID: 236be3fb335142d6a4092ad9c8733b3a
I20260203 21:07:32.469794 8509 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:07:32.470296 8509 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:07:32.470674 8509 catalog_manager.cc:6038] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: Loaded TSK: 0
I20260203 21:07:32.471330 8509 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260203 21:07:32.551222 7773 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260203 21:07:32.553530 7773 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:07:32.556284 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.556051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:59908)
0203 21:07:32.556166 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.556213 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260203 21:07:32.556274 7966 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [term 3 FOLLOWER]: Refusing update from remote peer c31601e9963b484bbb7869e10d137467: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260203 21:07:32.556747 8504 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260203 21:07:32.557093 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.556846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.0.1:59914)
0203 21:07:32.556968 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.557023 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260203 21:07:32.557205 7758 consensus_peers.cc:597] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 -> Peer 61f5416a4138441e8bdfa4f27aeb963d (127.0.102.190:42297): Couldn't send request to peer 61f5416a4138441e8bdfa4f27aeb963d. Status: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260203 21:07:32.558300 8511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c31601e9963b484bbb7869e10d137467. Latest consensus state: current_term: 3 leader_uuid: "c31601e9963b484bbb7869e10d137467" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } } }
I20260203 21:07:32.558415 8511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [sys.catalog]: This master's current role is: FOLLOWER
I20260203 21:07:32.559617 8504 sys_catalog.cc:455] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "c31601e9963b484bbb7869e10d137467" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } } }
I20260203 21:07:32.559733 8504 sys_catalog.cc:458] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467 [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:32.565177 8511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "c31601e9963b484bbb7869e10d137467" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61f5416a4138441e8bdfa4f27aeb963d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 42297 } } peers { permanent_uuid: "c31601e9963b484bbb7869e10d137467" member_type: VOTER last_known_addr { host: "127.0.102.189" port: 38693 } attrs { promote: false } } peers { permanent_uuid: "0ef1705ecf404c62b8d1ca281e959679" member_type: VOTER last_known_addr { host: "127.0.102.188" port: 45925 } attrs { promote: false } } }
I20260203 21:07:32.565404 8511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ef1705ecf404c62b8d1ca281e959679 [sys.catalog]: This master's current role is: FOLLOWER
I20260203 21:07:32.572477 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.568697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:34113 (local address 127.0.0.1:55936)
0203 21:07:32.569223 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.569237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:32.569336 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:32.569690 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:32.569694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:32.569715 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:32.569993 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:32.570002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.571172 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:32.571177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:32.572105 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:32.572116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.572210 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:32.572235 (+ 25us) client_negotiation.cc:769] Sending connection context
0203 21:07:32.572280 (+ 45us) client_negotiation.cc:240] Negotiation successful
0203 21:07:32.572343 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458}
I20260203 21:07:32.573364 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.568994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:36543 (local address 127.0.0.1:57058)
0203 21:07:32.569942 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.569954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:32.570028 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:32.570577 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:32.570580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:32.570594 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:32.570786 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:32.570792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.571925 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:32.571929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:32.573032 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:32.573041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.573133 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:32.573149 (+ 16us) client_negotiation.cc:769] Sending connection context
0203 21:07:32.573194 (+ 45us) client_negotiation.cc:240] Negotiation successful
0203 21:07:32.573247 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"thread_start_us":85,"threads_started":1}
I20260203 21:07:32.573683 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.568821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.0.102.131:34113)
0203 21:07:32.569172 (+ 351us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:32.569178 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:32.569404 (+ 226us) server_negotiation.cc:408] Connection header received
0203 21:07:32.569468 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:32.569472 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:32.569530 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:32.570120 (+ 590us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:32.570234 (+ 114us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.571023 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:32.572984 (+ 1961us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.573361 (+ 377us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:32.573395 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:32.573450 (+ 55us) server_negotiation.cc:300] Negotiation successful
0203 21:07:32.573516 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":93,"threads_started":1}
I20260203 21:07:32.573714 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.569616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37584 (local address 127.0.102.130:45287)
0203 21:07:32.569891 (+ 275us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:32.569896 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:32.570241 (+ 345us) server_negotiation.cc:408] Connection header received
0203 21:07:32.570358 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:32.570362 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:32.570408 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:32.570478 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:32.570875 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.571664 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:32.572809 (+ 1145us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.573273 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:32.573309 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:32.573516 (+ 207us) server_negotiation.cc:300] Negotiation successful
0203 21:07:32.573587 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":71,"threads_started":1}
I20260203 21:07:32.574249 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.569102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57058 (local address 127.0.102.129:36543)
0203 21:07:32.570038 (+ 936us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:32.570043 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:32.570065 (+ 22us) server_negotiation.cc:408] Connection header received
0203 21:07:32.570114 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:32.570117 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:32.570166 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:32.570880 (+ 714us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:32.570979 (+ 99us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.571825 (+ 846us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:32.573516 (+ 1691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.573997 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:32.574031 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:32.574076 (+ 45us) server_negotiation.cc:300] Negotiation successful
0203 21:07:32.574133 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":78,"threads_started":1}
I20260203 21:07:32.574203 8356 tablet_service.cc:1505] Processing CreateTablet for tablet c99fdcc301784dd3a506fe60eb31a81f (DEFAULT_TABLE table=test-table [id=47ae1de0d2e1425498be27a904ad8366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:32.574488 8356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c99fdcc301784dd3a506fe60eb31a81f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:32.574731 8085 tablet_service.cc:1505] Processing CreateTablet for tablet c99fdcc301784dd3a506fe60eb31a81f (DEFAULT_TABLE table=test-table [id=47ae1de0d2e1425498be27a904ad8366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:32.574935 8085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c99fdcc301784dd3a506fe60eb31a81f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:32.576740 8521 tablet_bootstrap.cc:492] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f: Bootstrap starting.
I20260203 21:07:32.576789 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.569356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:45287 (local address 127.0.0.1:37584)
0203 21:07:32.570177 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.570188 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:32.570263 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:32.570492 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:32.570495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:32.570510 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:32.570747 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:32.570754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.571801 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:32.571804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:32.572665 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:32.572675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.573377 (+ 702us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:32.573391 (+ 14us) client_negotiation.cc:769] Sending connection context
0203 21:07:32.573881 (+ 490us) client_negotiation.cc:240] Negotiation successful
0203 21:07:32.576673 (+ 2792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":88,"threads_started":1}
I20260203 21:07:32.577131 8522 tablet_bootstrap.cc:492] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178: Bootstrap starting.
I20260203 21:07:32.577263 8221 tablet_service.cc:1505] Processing CreateTablet for tablet c99fdcc301784dd3a506fe60eb31a81f (DEFAULT_TABLE table=test-table [id=47ae1de0d2e1425498be27a904ad8366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:32.577508 8221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c99fdcc301784dd3a506fe60eb31a81f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:32.577721 8521 tablet_bootstrap.cc:654] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:32.577971 8522 tablet_bootstrap.cc:654] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:32.579030 8522 tablet_bootstrap.cc:492] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178: No bootstrap required, opened a new log
I20260203 21:07:32.579115 8522 ts_tablet_manager.cc:1403] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260203 21:07:32.579563 8522 raft_consensus.cc:359] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } } peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } }
I20260203 21:07:32.579682 8522 raft_consensus.cc:385] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:32.579723 8522 raft_consensus.cc:740] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1f9e3fc5a4e44edaca392a6e083f178, State: Initialized, Role: FOLLOWER
I20260203 21:07:32.579811 8525 tablet_bootstrap.cc:492] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Bootstrap starting.
I20260203 21:07:32.579877 8522 consensus_queue.cc:260] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } } peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } }
I20260203 21:07:32.580219 8522 ts_tablet_manager.cc:1434] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:32.580524 8521 tablet_bootstrap.cc:492] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f: No bootstrap required, opened a new log
I20260203 21:07:32.580768 8521 ts_tablet_manager.cc:1403] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260203 21:07:32.580812 8525 tablet_bootstrap.cc:654] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:32.580940 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.580573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.129:37147)
0203 21:07:32.580826 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.580875 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260203 21:07:32.581389 8521 raft_consensus.cc:359] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } } peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } }
I20260203 21:07:32.581543 8521 raft_consensus.cc:385] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:32.581600 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.581310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.129:57385)
0203 21:07:32.581464 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.581517 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260203 21:07:32.581619 8521 raft_consensus.cc:740] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8644652b1a349dbbb003fe66b88e94f, State: Initialized, Role: FOLLOWER
I20260203 21:07:32.581840 8521 consensus_queue.cc:260] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } } peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } }
I20260203 21:07:32.582149 8521 ts_tablet_manager.cc:1434] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:32.582774 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.582434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.131:33489)
0203 21:07:32.582662 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.582710 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260203 21:07:32.583484 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.583175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.131:46297)
0203 21:07:32.583382 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.583427 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260203 21:07:32.583516 8153 heartbeater.cc:499] Master 127.0.102.189:38693 was elected leader, sending a full tablet report...
I20260203 21:07:32.584234 8525 tablet_bootstrap.cc:492] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: No bootstrap required, opened a new log
I20260203 21:07:32.584372 8525 ts_tablet_manager.cc:1403] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260203 21:07:32.584885 8525 raft_consensus.cc:359] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } } peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } }
I20260203 21:07:32.585083 8525 raft_consensus.cc:385] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:32.585119 8525 raft_consensus.cc:740] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5f9dfb6e7a4fc0bb51f68658d4b05f, State: Initialized, Role: FOLLOWER
I20260203 21:07:32.585240 8525 consensus_queue.cc:260] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } } peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } }
I20260203 21:07:32.585577 8525 ts_tablet_manager.cc:1434] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:32.586053 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.585849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.130:55033)
0203 21:07:32.585946 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.585987 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260203 21:07:32.586645 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.586420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.130:54999)
0203 21:07:32.586520 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.586569 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260203 21:07:32.873905 8529 raft_consensus.cc:493] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:07:32.874037 8529 raft_consensus.cc:515] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } } peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } }
I20260203 21:07:32.875095 8529 leader_election.cc:290] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8644652b1a349dbbb003fe66b88e94f (127.0.102.131:34113), f1f9e3fc5a4e44edaca392a6e083f178 (127.0.102.129:36543)
I20260203 21:07:32.878294 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.875345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:34113 (local address 127.0.102.130:45595)
0203 21:07:32.875534 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.875551 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:32.875686 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:32.876080 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:32.876084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:32.876104 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:32.876390 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:32.876399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.877162 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:32.877241 (+ 79us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:32.877861 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:32.877869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.878006 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:32.878025 (+ 19us) client_negotiation.cc:769] Sending connection context
0203 21:07:32.878075 (+ 50us) client_negotiation.cc:240] Negotiation successful
0203 21:07:32.878148 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260203 21:07:32.878309 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.875650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:36543 (local address 127.0.102.130:60023)
0203 21:07:32.875994 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.876006 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:32.876095 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:32.876312 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:32.876314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:32.876326 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:32.876521 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:32.876526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.877246 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:32.877248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:32.877851 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:32.877859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.877967 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:32.877991 (+ 24us) client_negotiation.cc:769] Sending connection context
0203 21:07:32.878040 (+ 49us) client_negotiation.cc:240] Negotiation successful
0203 21:07:32.878097 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":130,"threads_started":1}
I20260203 21:07:32.878633 8528 raft_consensus.cc:493] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:07:32.878702 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.875778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:60023 (local address 127.0.102.129:36543)
0203 21:07:32.875996 (+ 218us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:32.876001 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:32.876071 (+ 70us) server_negotiation.cc:408] Connection header received
0203 21:07:32.876178 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:32.876181 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:32.876229 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:32.876306 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:32.876666 (+ 360us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.877134 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:32.877966 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.878440 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:32.878482 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:32.878534 (+ 52us) server_negotiation.cc:300] Negotiation successful
0203 21:07:32.878592 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260203 21:07:32.878762 8528 raft_consensus.cc:515] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } } peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } }
I20260203 21:07:32.878902 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.875427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:45595 (local address 127.0.102.131:34113)
0203 21:07:32.875576 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:32.875581 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:32.875635 (+ 54us) server_negotiation.cc:408] Connection header received
0203 21:07:32.875830 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:32.875834 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:32.875889 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:32.875963 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:32.876501 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.877060 (+ 559us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:32.877994 (+ 934us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.878456 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:32.878496 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:32.878548 (+ 52us) server_negotiation.cc:300] Negotiation successful
0203 21:07:32.878607 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260203 21:07:32.879148 8105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c99fdcc301784dd3a506fe60eb31a81f" candidate_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" is_pre_election: true
I20260203 21:07:32.879276 8377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c99fdcc301784dd3a506fe60eb31a81f" candidate_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8644652b1a349dbbb003fe66b88e94f" is_pre_election: true
I20260203 21:07:32.879374 8105 raft_consensus.cc:2468] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e5f9dfb6e7a4fc0bb51f68658d4b05f in term 0.
I20260203 21:07:32.879777 8377 raft_consensus.cc:2468] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e5f9dfb6e7a4fc0bb51f68658d4b05f in term 0.
I20260203 21:07:32.879777 8176 leader_election.cc:304] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e5f9dfb6e7a4fc0bb51f68658d4b05f, f1f9e3fc5a4e44edaca392a6e083f178; no voters:
I20260203 21:07:32.880024 8529 raft_consensus.cc:2804] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:07:32.880091 8529 raft_consensus.cc:493] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:07:32.880129 8529 raft_consensus.cc:3060] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:32.880504 8528 leader_election.cc:290] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1f9e3fc5a4e44edaca392a6e083f178 (127.0.102.129:36543), 2e5f9dfb6e7a4fc0bb51f68658d4b05f (127.0.102.130:45287)
I20260203 21:07:32.881321 8529 raft_consensus.cc:515] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } } peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } }
I20260203 21:07:32.881688 8529 leader_election.cc:290] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [CANDIDATE]: Term 1 election: Requested vote from peers a8644652b1a349dbbb003fe66b88e94f (127.0.102.131:34113), f1f9e3fc5a4e44edaca392a6e083f178 (127.0.102.129:36543)
I20260203 21:07:32.882010 8377 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c99fdcc301784dd3a506fe60eb31a81f" candidate_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8644652b1a349dbbb003fe66b88e94f"
I20260203 21:07:32.882113 8377 raft_consensus.cc:3060] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:32.883124 8377 raft_consensus.cc:2468] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e5f9dfb6e7a4fc0bb51f68658d4b05f in term 1.
I20260203 21:07:32.883365 8176 leader_election.cc:304] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e5f9dfb6e7a4fc0bb51f68658d4b05f, a8644652b1a349dbbb003fe66b88e94f; no voters:
I20260203 21:07:32.883694 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.880785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:45287 (local address 127.0.102.131:34361)
0203 21:07:32.881081 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.881095 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:32.881179 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:32.881570 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:32.881573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:32.881588 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:32.881816 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:32.881823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.882541 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:32.882545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:32.883092 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:32.883099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.883360 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:32.883391 (+ 31us) client_negotiation.cc:769] Sending connection context
0203 21:07:32.883510 (+ 119us) client_negotiation.cc:240] Negotiation successful
0203 21:07:32.883569 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":122,"threads_started":1}
I20260203 21:07:32.883736 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.880727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:36543 (local address 127.0.102.131:41597)
0203 21:07:32.880830 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.880844 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:32.880931 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:32.881375 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:32.881379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:32.881396 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:32.881612 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:32.881619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.882585 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:32.882588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:32.883395 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:32.883403 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.883531 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:32.883549 (+ 18us) client_negotiation.cc:769] Sending connection context
0203 21:07:32.883595 (+ 46us) client_negotiation.cc:240] Negotiation successful
0203 21:07:32.883647 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260203 21:07:32.883860 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.881233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:34361 (local address 127.0.102.130:45287)
0203 21:07:32.881341 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:32.881345 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:32.881357 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:32.881420 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:32.881423 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:32.881467 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:32.881536 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:32.881913 (+ 377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.882435 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:32.883199 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.883641 (+ 442us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:32.883674 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:32.883721 (+ 47us) server_negotiation.cc:300] Negotiation successful
0203 21:07:32.883770 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260203 21:07:32.883987 8529 raft_consensus.cc:2804] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:32.884169 8529 raft_consensus.cc:697] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 1 LEADER]: Becoming Leader. State: Replica: 2e5f9dfb6e7a4fc0bb51f68658d4b05f, State: Running, Role: LEADER
I20260203 21:07:32.884183 8105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c99fdcc301784dd3a506fe60eb31a81f" candidate_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f9e3fc5a4e44edaca392a6e083f178"
I20260203 21:07:32.884305 8105 raft_consensus.cc:3060] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:32.884343 8241 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c99fdcc301784dd3a506fe60eb31a81f" candidate_uuid: "a8644652b1a349dbbb003fe66b88e94f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" is_pre_election: true
I20260203 21:07:32.884435 8289 heartbeater.cc:499] Master 127.0.102.189:38693 was elected leader, sending a full tablet report...
I20260203 21:07:32.884411 8529 consensus_queue.cc:237] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } } peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } }
I20260203 21:07:32.885462 8105 raft_consensus.cc:2468] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e5f9dfb6e7a4fc0bb51f68658d4b05f in term 1.
I20260203 21:07:32.885564 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.881066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:41597 (local address 127.0.102.129:36543)
0203 21:07:32.881174 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:32.881178 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:32.881190 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:32.881228 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:32.881231 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:32.881276 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:32.881345 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:32.881816 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.882447 (+ 631us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:32.884848 (+ 2401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.885272 (+ 424us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:32.885309 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:32.885350 (+ 41us) server_negotiation.cc:300] Negotiation successful
0203 21:07:32.885416 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260203 21:07:32.885986 8105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c99fdcc301784dd3a506fe60eb31a81f" candidate_uuid: "a8644652b1a349dbbb003fe66b88e94f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" is_pre_election: true
I20260203 21:07:32.886138 8105 raft_consensus.cc:2393] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a8644652b1a349dbbb003fe66b88e94f in current term 1: Already voted for candidate 2e5f9dfb6e7a4fc0bb51f68658d4b05f in this term.
I20260203 21:07:32.886946 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.886709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.130:38971)
0203 21:07:32.886824 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.886882 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260203 21:07:32.886927 7772 catalog_manager.cc:5654] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e5f9dfb6e7a4fc0bb51f68658d4b05f (127.0.102.130). New cstate: current_term: 1 leader_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e5f9dfb6e7a4fc0bb51f68658d4b05f" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 45287 } health_report { overall_health: HEALTHY } } }
I20260203 21:07:32.887866 8312 leader_election.cc:304] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8644652b1a349dbbb003fe66b88e94f; no voters: 2e5f9dfb6e7a4fc0bb51f68658d4b05f, f1f9e3fc5a4e44edaca392a6e083f178
I20260203 21:07:32.888067 8528 raft_consensus.cc:2749] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260203 21:07:32.933634 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.930802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37588 (local address 127.0.102.130:45287)
0203 21:07:32.930929 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:32.930932 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:32.930946 (+ 14us) server_negotiation.cc:408] Connection header received
0203 21:07:32.931089 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:32.931091 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:32.931147 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:32.931217 (+ 70us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:07:32.931511 (+ 294us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.931980 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:32.932582 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:32.932752 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:32.932817 (+ 65us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:07:32.933300 (+ 483us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:07:32.933384 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:32.933457 (+ 73us) server_negotiation.cc:300] Negotiation successful
0203 21:07:32.933502 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260203 21:07:32.936400 8377 raft_consensus.cc:1275] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f [term 1 FOLLOWER]: Refusing update from remote peer 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:32.936443 8105 raft_consensus.cc:1275] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178 [term 1 FOLLOWER]: Refusing update from remote peer 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:32.936940 8532 consensus_queue.cc:1048] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8644652b1a349dbbb003fe66b88e94f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 34113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:32.937129 8529 consensus_queue.cc:1048] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1f9e3fc5a4e44edaca392a6e083f178" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 36543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:32.938313 8425 heartbeater.cc:499] Master 127.0.102.189:38693 was elected leader, sending a full tablet report...
I20260203 21:07:32.938791 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.938480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.131:43289)
0203 21:07:32.938644 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.938711 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260203 21:07:32.939451 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.939035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.129:53323)
0203 21:07:32.939304 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.939371 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260203 21:07:32.942739 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.942345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.130:57957)
0203 21:07:32.942622 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.942669 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260203 21:07:32.943514 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.943091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.131:45137)
0203 21:07:32.943393 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.943444 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226}
I20260203 21:07:32.944927 8538 mvcc.cc:204] Tried to move back new op lower bound from 7250546085622161408 to 7250546085417254912. Current Snapshot: MvccSnapshot[applied={T|T < 7250546085622161408 or (T in {7250546085622161408})}]
I20260203 21:07:32.945020 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.944749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.129:37723)
0203 21:07:32.944898 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.944947 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260203 21:07:32.946595 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:32.946341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:42297 (local address 127.0.102.130:57029)
0203 21:07:32.946477 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.946532 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260203 21:07:32.951424 7772 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:07:32.951586 7772 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:07:32.955315 7772 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c31601e9963b484bbb7869e10d137467: Sending DeleteTablet for 3 replicas of tablet c99fdcc301784dd3a506fe60eb31a81f
I20260203 21:07:32.956174 8356 tablet_service.cc:1552] Processing DeleteTablet for tablet c99fdcc301784dd3a506fe60eb31a81f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:32 UTC) from {username='oryx', principal='oryx/127.0.102.189@KRBTEST.COM'} at 127.0.0.1:55936
I20260203 21:07:32.956403 8085 tablet_service.cc:1552] Processing DeleteTablet for tablet c99fdcc301784dd3a506fe60eb31a81f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:32 UTC) from {username='oryx', principal='oryx/127.0.102.189@KRBTEST.COM'} at 127.0.0.1:57058
I20260203 21:07:32.956875 8221 tablet_service.cc:1552] Processing DeleteTablet for tablet c99fdcc301784dd3a506fe60eb31a81f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:32 UTC) from {username='oryx', principal='oryx/127.0.102.189@KRBTEST.COM'} at 127.0.0.1:37584
I20260203 21:07:32.957094 8547 tablet_replica.cc:333] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178: stopping tablet replica
I20260203 21:07:32.957229 8547 raft_consensus.cc:2243] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:32.957373 8547 raft_consensus.cc:2272] T c99fdcc301784dd3a506fe60eb31a81f P f1f9e3fc5a4e44edaca392a6e083f178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:32.957551 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 8022
I20260203 21:07:32.957609 8546 tablet_replica.cc:333] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f: stopping tablet replica
I20260203 21:07:32.958216 8546 raft_consensus.cc:2243] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:32.958344 8546 raft_consensus.cc:2272] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:32.958709 8548 tablet_replica.cc:333] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: stopping tablet replica
I20260203 21:07:32.958858 8548 raft_consensus.cc:2243] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 1 LEADER]: Raft consensus shutting down.
I20260203 21:07:32.959074 8546 ts_tablet_manager.cc:1916] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:32.959291 8548 raft_consensus.cc:2272] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:32.959964 8548 ts_tablet_manager.cc:1916] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:32.961753 8546 ts_tablet_manager.cc:1929] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:07:32.961833 8546 log.cc:1199] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-2/wal/wals/c99fdcc301784dd3a506fe60eb31a81f
I20260203 21:07:32.962098 8546 ts_tablet_manager.cc:1950] T c99fdcc301784dd3a506fe60eb31a81f P a8644652b1a349dbbb003fe66b88e94f: Deleting consensus metadata
I20260203 21:07:32.962479 8548 ts_tablet_manager.cc:1929] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:07:32.962604 8548 log.cc:1199] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1770152788143875-410-0/minicluster-data/ts-1/wal/wals/c99fdcc301784dd3a506fe60eb31a81f
I20260203 21:07:32.962867 7759 catalog_manager.cc:4985] TS a8644652b1a349dbbb003fe66b88e94f (127.0.102.131:34113): tablet c99fdcc301784dd3a506fe60eb31a81f (table test-table [id=47ae1de0d2e1425498be27a904ad8366]) successfully deleted
I20260203 21:07:32.962935 8548 ts_tablet_manager.cc:1950] T c99fdcc301784dd3a506fe60eb31a81f P 2e5f9dfb6e7a4fc0bb51f68658d4b05f: Deleting consensus metadata
I20260203 21:07:32.963593 7757 catalog_manager.cc:4985] TS 2e5f9dfb6e7a4fc0bb51f68658d4b05f (127.0.102.130:45287): tablet c99fdcc301784dd3a506fe60eb31a81f (table test-table [id=47ae1de0d2e1425498be27a904ad8366]) successfully deleted
I20260203 21:07:32.968254 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 8158
W20260203 21:07:32.968387 8517 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:32.967926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:36543 (local address 127.0.0.1:57064)
0203 21:07:32.968239 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:32.968291 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.129:36543: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":223}
W20260203 21:07:32.971369 7759 catalog_manager.cc:4712] TS f1f9e3fc5a4e44edaca392a6e083f178 (127.0.102.129:36543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c99fdcc301784dd3a506fe60eb31a81f: Network error: Client connection negotiation failed: client connection to 127.0.102.129:36543: connect: Connection reset by peer (error 104)
I20260203 21:07:32.973505 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 8294
I20260203 21:07:32.978607 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 7634
I20260203 21:07:32.983301 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 7745
2026-02-03T21:07:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9976 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:07:33 dist-test-slave-j889 krb5kdc[8551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:07:33 dist-test-slave-j889 krb5kdc[8551](info): set up 2 sockets
Feb 03 21:07:33 dist-test-slave-j889 krb5kdc[8551](info): commencing operation
krb5kdc: starting...
W20260203 21:07:34.821139 410 mini_kdc.cc:121] Time spent starting KDC: real 1.816s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:07:34 dist-test-slave-j889 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:07:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:07:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab.
I20260203 21:07:34.946996 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:43097
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:44665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:43097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:35.051337 8567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:35.051564 8567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:35.051620 8567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:35.054982 8567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:07:35.055047 8567 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260203 21:07:35.055063 8567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:35.055084 8567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:35.055100 8567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:07:35.055119 8567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:07:35.059307 8567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:44665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:43097
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:43097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:35.060302 8567 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:35.061128 8567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:35.066612 8572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:35.066711 8567 server_base.cc:1060] running on GCE node
W20260203 21:07:35.066641 8573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:35.066883 8575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:35.067360 8567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:35.068225 8567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:35.069386 8567 hybrid_clock.cc:648] HybridClock initialized: now 1770152855069373 us; error 30 us; skew 500 ppm
Feb 03 21:07:35 dist-test-slave-j889 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:35.071858 8567 init.cc:377] Logged in from keytab as oryx/127.0.102.190@KRBTEST.COM (short username oryx)
I20260203 21:07:35.072897 8567 webserver.cc:492] Webserver started at http://127.0.102.190:34605/ using document root <none> and password file <none>
I20260203 21:07:35.073400 8567 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:35.073449 8567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:35.073618 8567 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:35.075263 8567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "21c8fb97ac064d7cb5c7d4828cfba713"
format_stamp: "Formatted at 2026-02-03 21:07:35 on dist-test-slave-j889"
server_key: "5ff33a9e5d6474dbb0d866cfdfb010c6"
server_key_iv: "39856b651b920d54d09fa23190ed6df9"
server_key_version: "encryptionkey@0"
I20260203 21:07:35.075709 8567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "21c8fb97ac064d7cb5c7d4828cfba713"
format_stamp: "Formatted at 2026-02-03 21:07:35 on dist-test-slave-j889"
server_key: "5ff33a9e5d6474dbb0d866cfdfb010c6"
server_key_iv: "39856b651b920d54d09fa23190ed6df9"
server_key_version: "encryptionkey@0"
I20260203 21:07:35.078888 8567 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260203 21:07:35.080964 8582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:35.081982 8567 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:35.082075 8567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "21c8fb97ac064d7cb5c7d4828cfba713"
format_stamp: "Formatted at 2026-02-03 21:07:35 on dist-test-slave-j889"
server_key: "5ff33a9e5d6474dbb0d866cfdfb010c6"
server_key_iv: "39856b651b920d54d09fa23190ed6df9"
server_key_version: "encryptionkey@0"
I20260203 21:07:35.082166 8567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:35.097502 8567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:35.100136 8567 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:35.100292 8567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:35.107664 8567 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:43097
I20260203 21:07:35.107898 8644 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:43097 every 8 connection(s)
I20260203 21:07:35.108609 8567 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:07:35.111176 8645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:35.111686 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 8567
I20260203 21:07:35.111779 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:07:35.112005 410 external_mini_cluster.cc:1468] Setting key 75d910b4774e5ef19af24ce5f59a3aec
I20260203 21:07:35.116863 8645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713: Bootstrap starting.
Feb 03 21:07:35 dist-test-slave-j889 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM
I20260203 21:07:35.118940 8645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:35.119649 8645 log.cc:826] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:35.121583 8645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713: No bootstrap required, opened a new log
I20260203 21:07:35.123594 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.113105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39934 (local address 127.0.102.190:43097)
0203 21:07:35.113494 (+ 389us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:35.113506 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:35.113543 (+ 37us) server_negotiation.cc:408] Connection header received
0203 21:07:35.114200 (+ 657us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:35.114223 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:35.114564 (+ 341us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:35.114880 (+ 316us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:35.115593 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.116733 (+ 1140us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.117360 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.117607 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.119411 (+ 1804us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:35.119430 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:35.119443 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:35.119479 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:35.121384 (+ 1905us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:35.121764 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:35.121768 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:35.121773 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:35.121840 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:35.122053 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:35.122056 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:35.122059 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:35.122358 (+ 299us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:35.122556 (+ 198us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:35.122818 (+ 262us) server_negotiation.cc:300] Negotiation successful
0203 21:07:35.123037 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
I20260203 21:07:35.124395 8645 raft_consensus.cc:359] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c8fb97ac064d7cb5c7d4828cfba713" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43097 } }
I20260203 21:07:35.124572 8645 raft_consensus.cc:385] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:35.124646 8645 raft_consensus.cc:740] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c8fb97ac064d7cb5c7d4828cfba713, State: Initialized, Role: FOLLOWER
I20260203 21:07:35.125039 8645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c8fb97ac064d7cb5c7d4828cfba713" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43097 } }
I20260203 21:07:35.125159 8645 raft_consensus.cc:399] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:07:35.125216 8645 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:07:35.125303 8645 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:35.126170 8645 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c8fb97ac064d7cb5c7d4828cfba713" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43097 } }
I20260203 21:07:35.126477 8645 leader_election.cc:304] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21c8fb97ac064d7cb5c7d4828cfba713; no voters:
I20260203 21:07:35.126729 8645 leader_election.cc:290] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:07:35.126814 8650 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:35.127017 8650 raft_consensus.cc:697] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [term 1 LEADER]: Becoming Leader. State: Replica: 21c8fb97ac064d7cb5c7d4828cfba713, State: Running, Role: LEADER
I20260203 21:07:35.127341 8650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c8fb97ac064d7cb5c7d4828cfba713" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43097 } }
I20260203 21:07:35.127794 8645 sys_catalog.cc:565] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:07:35.128795 8651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21c8fb97ac064d7cb5c7d4828cfba713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c8fb97ac064d7cb5c7d4828cfba713" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43097 } } }
I20260203 21:07:35.128899 8651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:35.129163 8652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21c8fb97ac064d7cb5c7d4828cfba713. Latest consensus state: current_term: 1 leader_uuid: "21c8fb97ac064d7cb5c7d4828cfba713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21c8fb97ac064d7cb5c7d4828cfba713" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43097 } } }
I20260203 21:07:35.129238 8659 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:07:35.129299 8652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713 [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:35.131700 8659 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:07:35.136185 8659 catalog_manager.cc:1357] Generated new cluster ID: 410e59feac5d4a95b98881c722222d9f
I20260203 21:07:35.136246 8659 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:07:35.145498 8659 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:07:35.146328 8659 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:07:35.154035 8659 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713: Generated new TSK 0
I20260203 21:07:35.154682 8659 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab.
I20260203 21:07:35.194967 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:43097
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:44665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:35.298695 8673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:35.298954 8673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:35.299026 8673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:35.302253 8673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:35.302309 8673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:35.302366 8673 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260203 21:07:35.302426 8673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:07:35.306511 8673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:44665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.102.190:43097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:35.307511 8673 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:35.308337 8673 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:35.314880 8679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:35.314939 8673 server_base.cc:1060] running on GCE node
W20260203 21:07:35.314815 8681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:35.314944 8678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:35.315438 8673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:35.315941 8673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:35.317076 8673 hybrid_clock.cc:648] HybridClock initialized: now 1770152855317066 us; error 25 us; skew 500 ppm
Feb 03 21:07:35 dist-test-slave-j889 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:35.319478 8673 init.cc:377] Logged in from keytab as oryx/127.0.102.129@KRBTEST.COM (short username oryx)
I20260203 21:07:35.320456 8673 webserver.cc:492] Webserver started at http://127.0.102.129:42107/ using document root <none> and password file <none>
I20260203 21:07:35.321019 8673 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:35.321069 8673 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:35.321233 8673 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:35.322827 8673 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "cfcee84ea65d44cebf446d3ab030ed42"
format_stamp: "Formatted at 2026-02-03 21:07:35 on dist-test-slave-j889"
server_key: "d428be2a04e84880eecaa8b68a6d0e57"
server_key_iv: "e04844d86e3b1b160271f54cbf89c506"
server_key_version: "encryptionkey@0"
I20260203 21:07:35.323271 8673 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "cfcee84ea65d44cebf446d3ab030ed42"
format_stamp: "Formatted at 2026-02-03 21:07:35 on dist-test-slave-j889"
server_key: "d428be2a04e84880eecaa8b68a6d0e57"
server_key_iv: "e04844d86e3b1b160271f54cbf89c506"
server_key_version: "encryptionkey@0"
I20260203 21:07:35.326609 8673 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260203 21:07:35.328711 8688 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:35.329746 8673 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:35.329842 8673 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "cfcee84ea65d44cebf446d3ab030ed42"
format_stamp: "Formatted at 2026-02-03 21:07:35 on dist-test-slave-j889"
server_key: "d428be2a04e84880eecaa8b68a6d0e57"
server_key_iv: "e04844d86e3b1b160271f54cbf89c506"
server_key_version: "encryptionkey@0"
I20260203 21:07:35.329924 8673 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:35.350545 8673 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:35.353307 8673 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:35.353479 8673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:35.354852 8673 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:35.354900 8673 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:35.354951 8673 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:35.354979 8673 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 03 21:07:35 dist-test-slave-j889 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM
I20260203 21:07:35.364964 8673 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:37411
I20260203 21:07:35.364991 8803 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:37411 every 8 connection(s)
I20260203 21:07:35.365938 8673 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:07:35.366545 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.355544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:36209 (local address 127.0.102.190:43097)
0203 21:07:35.355662 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:35.355666 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:35.356509 (+ 843us) server_negotiation.cc:408] Connection header received
0203 21:07:35.357441 (+ 932us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:35.357445 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:35.357505 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:35.357586 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:35.359206 (+ 1620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.360038 (+ 832us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.360899 (+ 861us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.361082 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.363216 (+ 2134us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:35.363238 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:35.363241 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:35.363274 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:35.364590 (+ 1316us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:35.365146 (+ 556us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:35.365149 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:35.365151 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:35.365199 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:35.365459 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:35.365462 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:35.365464 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:35.365627 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:35.365712 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:35.366234 (+ 522us) server_negotiation.cc:300] Negotiation successful
0203 21:07:35.366390 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260203 21:07:35.367180 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.355881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:43097 (local address 127.0.102.129:36209)
0203 21:07:35.356366 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:35.356402 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:35.357198 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:35.357772 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:35.357782 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:35.358305 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:35.359067 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.359081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.360154 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.360158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:35.360793 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.360803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.361005 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.361571 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:35.361594 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:35.363079 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:35.364769 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:35.364776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:35.364788 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:35.365034 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:35.365281 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:35.365284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:35.365287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:35.365377 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:35.365722 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:35.365728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:35.365979 (+ 251us) client_negotiation.cc:769] Sending connection context
0203 21:07:35.366178 (+ 199us) client_negotiation.cc:240] Negotiation successful
0203 21:07:35.366450 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":137,"threads_started":1}
I20260203 21:07:35.368295 8804 heartbeater.cc:344] Connected to a master server at 127.0.102.190:43097
I20260203 21:07:35.368443 8804 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:35.368966 8804 heartbeater.cc:507] Master 127.0.102.190:43097 requested a full tablet report, sending...
I20260203 21:07:35.370048 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 8673
I20260203 21:07:35.370163 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:07:35.370421 410 external_mini_cluster.cc:1468] Setting key fe0294002ec262aac4e0829ca047247d
I20260203 21:07:35.370558 8599 ts_manager.cc:194] Registered new tserver with Master: cfcee84ea65d44cebf446d3ab030ed42 (127.0.102.129:37411)
I20260203 21:07:35.372062 8599 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:36209
I20260203 21:07:35.376943 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.370523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:41345 (local address 127.0.102.190:43097)
0203 21:07:35.370634 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:35.370638 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:35.370649 (+ 11us) server_negotiation.cc:408] Connection header received
0203 21:07:35.370674 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:35.370677 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:35.370709 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:35.370772 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:35.371597 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.372314 (+ 717us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.373119 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.373268 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.374289 (+ 1021us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:35.374305 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:35.374309 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:35.374333 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:35.375630 (+ 1297us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:35.376049 (+ 419us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:35.376052 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:35.376053 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:35.376087 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:35.376288 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:35.376290 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:35.376291 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:35.376404 (+ 113us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:35.376466 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:35.376732 (+ 266us) server_negotiation.cc:300] Negotiation successful
0203 21:07:35.376837 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260203 21:07:35.377014 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.370211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:43097 (local address 127.0.102.129:41345)
0203 21:07:35.370448 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:35.370462 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:35.370546 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:35.370878 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:35.370881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:35.371059 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:35.371467 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.371475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.372422 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.372426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:35.372984 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.372995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.373358 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.373829 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:35.373845 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:35.374173 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:35.375737 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:35.375740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:35.375743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:35.375962 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:35.376156 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:35.376160 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:35.376162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:35.376217 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:35.376476 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:35.376479 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:35.376568 (+ 89us) client_negotiation.cc:769] Sending connection context
0203 21:07:35.376697 (+ 129us) client_negotiation.cc:240] Negotiation successful
0203 21:07:35.376838 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for oryx/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab.
I20260203 21:07:35.412998 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:43097
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:44665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:35.516443 8815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:35.516655 8815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:35.516733 8815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:35.519901 8815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:35.519953 8815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:35.520004 8815 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260203 21:07:35.520063 8815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:07:35.524112 8815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:44665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.102.190:43097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:35.525194 8815 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:35.526037 8815 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:35.532414 8820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:35.532516 8815 server_base.cc:1060] running on GCE node
W20260203 21:07:35.532423 8823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:35.532408 8821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:35.533018 8815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:35.533526 8815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:35.534791 8815 hybrid_clock.cc:648] HybridClock initialized: now 1770152855534772 us; error 28 us; skew 500 ppm
Feb 03 21:07:35 dist-test-slave-j889 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:35.537221 8815 init.cc:377] Logged in from keytab as oryx/127.0.102.130@KRBTEST.COM (short username oryx)
I20260203 21:07:35.538149 8815 webserver.cc:492] Webserver started at http://127.0.102.130:32867/ using document root <none> and password file <none>
I20260203 21:07:35.538635 8815 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:35.538681 8815 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:35.538851 8815 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:35.540457 8815 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "6e95102e3672465d9df500793a178c72"
format_stamp: "Formatted at 2026-02-03 21:07:35 on dist-test-slave-j889"
server_key: "6978442741e725d07d703ce41ad7fbfe"
server_key_iv: "99101355a5f44e79719801d5c6059a0f"
server_key_version: "encryptionkey@0"
I20260203 21:07:35.540916 8815 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "6e95102e3672465d9df500793a178c72"
format_stamp: "Formatted at 2026-02-03 21:07:35 on dist-test-slave-j889"
server_key: "6978442741e725d07d703ce41ad7fbfe"
server_key_iv: "99101355a5f44e79719801d5c6059a0f"
server_key_version: "encryptionkey@0"
I20260203 21:07:35.543926 8815 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260203 21:07:35.545933 8830 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:35.546869 8815 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260203 21:07:35.546967 8815 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "6e95102e3672465d9df500793a178c72"
format_stamp: "Formatted at 2026-02-03 21:07:35 on dist-test-slave-j889"
server_key: "6978442741e725d07d703ce41ad7fbfe"
server_key_iv: "99101355a5f44e79719801d5c6059a0f"
server_key_version: "encryptionkey@0"
I20260203 21:07:35.547051 8815 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:35.564013 8815 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:35.566676 8815 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:35.566872 8815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:35.568364 8815 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:35.568416 8815 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:35.568470 8815 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:35.568516 8815 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 03 21:07:35 dist-test-slave-j889 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM
I20260203 21:07:35.578852 8815 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:35873
I20260203 21:07:35.578872 8945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:35873 every 8 connection(s)
I20260203 21:07:35.579861 8815 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:07:35.579883 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.569017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:54919 (local address 127.0.102.190:43097)
0203 21:07:35.569150 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:35.569153 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:35.569968 (+ 815us) server_negotiation.cc:408] Connection header received
0203 21:07:35.570797 (+ 829us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:35.570802 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:35.570858 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:35.570930 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:35.572618 (+ 1688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.573179 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.573805 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.573980 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.576471 (+ 2491us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:35.576494 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:35.576498 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:35.576531 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:35.578083 (+ 1552us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:35.578510 (+ 427us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:35.578513 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:35.578515 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:35.578561 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:35.578868 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:35.578871 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:35.578872 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:35.579033 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:35.579127 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:35.579604 (+ 477us) server_negotiation.cc:300] Negotiation successful
0203 21:07:35.579724 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260203 21:07:35.580498 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.569379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:43097 (local address 127.0.102.130:54919)
0203 21:07:35.569836 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:35.569872 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:35.570608 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:35.571099 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:35.571107 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:35.571683 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:35.572470 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.572488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.573293 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.573297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:35.573700 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.573707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.573921 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.574498 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:35.574538 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:35.576320 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:35.578188 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:35.578193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:35.578204 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:35.578421 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:35.578659 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:35.578662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:35.578663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:35.578769 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:35.579136 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:35.579143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:35.579366 (+ 223us) client_negotiation.cc:769] Sending connection context
0203 21:07:35.579543 (+ 177us) client_negotiation.cc:240] Negotiation successful
0203 21:07:35.579819 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":129,"threads_started":1}
I20260203 21:07:35.581676 8946 heartbeater.cc:344] Connected to a master server at 127.0.102.190:43097
I20260203 21:07:35.581830 8946 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:35.582305 8946 heartbeater.cc:507] Master 127.0.102.190:43097 requested a full tablet report, sending...
I20260203 21:07:35.583243 8596 ts_manager.cc:194] Registered new tserver with Master: 6e95102e3672465d9df500793a178c72 (127.0.102.130:35873)
I20260203 21:07:35.583860 8596 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.102.130@KRBTEST.COM'} at 127.0.102.130:54919
I20260203 21:07:35.588124 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 8815
I20260203 21:07:35.588240 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:07:35.588459 410 external_mini_cluster.cc:1468] Setting key 43526e0d6bcd0ffa575a16ce30fdd1d4
I20260203 21:07:35.588610 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.583463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:55397 (local address 127.0.102.190:43097)
0203 21:07:35.583564 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:35.583568 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:35.583718 (+ 150us) server_negotiation.cc:408] Connection header received
0203 21:07:35.583817 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:35.583820 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:35.583861 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:35.583920 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:35.584439 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.584939 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.585497 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.585663 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.586356 (+ 693us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:35.586370 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:35.586373 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:35.586393 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:35.587359 (+ 966us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:35.587682 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:35.587685 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:35.587686 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:35.587722 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:35.587910 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:35.587913 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:35.587916 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:35.588062 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:35.588143 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:35.588397 (+ 254us) server_negotiation.cc:300] Negotiation successful
0203 21:07:35.588487 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260203 21:07:35.588652 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.583420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:43097 (local address 127.0.102.130:55397)
0203 21:07:35.583651 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:35.583662 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:35.583745 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:35.583943 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:35.583946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:35.584131 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:35.584344 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.584350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.585032 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.585034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:35.585405 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.585411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.585504 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.585939 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:35.585950 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:35.586249 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:35.587449 (+ 1200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:35.587453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:35.587455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:35.587602 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:35.587792 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:35.587796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:35.587798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:35.587840 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:35.588143 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:35.588147 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:35.588219 (+ 72us) client_negotiation.cc:769] Sending connection context
0203 21:07:35.588346 (+ 127us) client_negotiation.cc:240] Negotiation successful
0203 21:07:35.588460 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for oryx/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab.
I20260203 21:07:35.628575 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:43097
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:44665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:35.730491 8957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:35.730707 8957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:35.730779 8957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:35.734125 8957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:35.734189 8957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:35.734256 8957 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260203 21:07:35.734319 8957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:07:35.738509 8957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:44665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.102.190:43097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:35.739535 8957 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:35.740370 8957 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:35.746634 8965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:35.746675 8963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:35.746634 8962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:35.746795 8957 server_base.cc:1060] running on GCE node
I20260203 21:07:35.747251 8957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:35.747740 8957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:35.748878 8957 hybrid_clock.cc:648] HybridClock initialized: now 1770152855748866 us; error 26 us; skew 500 ppm
Feb 03 21:07:35 dist-test-slave-j889 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:35.751250 8957 init.cc:377] Logged in from keytab as oryx/127.0.102.131@KRBTEST.COM (short username oryx)
I20260203 21:07:35.752220 8957 webserver.cc:492] Webserver started at http://127.0.102.131:35617/ using document root <none> and password file <none>
I20260203 21:07:35.752749 8957 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:35.752797 8957 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:35.752950 8957 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:35.754541 8957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "fbfe1cf326d44927abd9f87b4c1daa09"
format_stamp: "Formatted at 2026-02-03 21:07:35 on dist-test-slave-j889"
server_key: "3368a00fc25f8c40f0d5e1f5982cc0b3"
server_key_iv: "9d835876285d664ae7e2b82c9da81fc1"
server_key_version: "encryptionkey@0"
I20260203 21:07:35.754947 8957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "fbfe1cf326d44927abd9f87b4c1daa09"
format_stamp: "Formatted at 2026-02-03 21:07:35 on dist-test-slave-j889"
server_key: "3368a00fc25f8c40f0d5e1f5982cc0b3"
server_key_iv: "9d835876285d664ae7e2b82c9da81fc1"
server_key_version: "encryptionkey@0"
I20260203 21:07:35.758056 8957 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260203 21:07:35.760016 8972 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:35.760984 8957 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260203 21:07:35.761085 8957 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "fbfe1cf326d44927abd9f87b4c1daa09"
format_stamp: "Formatted at 2026-02-03 21:07:35 on dist-test-slave-j889"
server_key: "3368a00fc25f8c40f0d5e1f5982cc0b3"
server_key_iv: "9d835876285d664ae7e2b82c9da81fc1"
server_key_version: "encryptionkey@0"
I20260203 21:07:35.761160 8957 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:35.788107 8957 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:35.790885 8957 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:35.791074 8957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:35.792491 8957 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:35.792539 8957 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:35.792591 8957 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:35.792619 8957 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 03 21:07:35 dist-test-slave-j889 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM
I20260203 21:07:35.802950 8957 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:43563
I20260203 21:07:35.803004 9087 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:43563 every 8 connection(s)
I20260203 21:07:35.803333 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.793036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:51145 (local address 127.0.102.190:43097)
0203 21:07:35.793193 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:35.793199 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:35.793904 (+ 705us) server_negotiation.cc:408] Connection header received
0203 21:07:35.794739 (+ 835us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:35.794742 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:35.794800 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:35.794876 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:35.796198 (+ 1322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.796885 (+ 687us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.797511 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.797679 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.800051 (+ 2372us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:35.800072 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:35.800076 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:35.800103 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:35.801430 (+ 1327us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:35.801886 (+ 456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:35.801889 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:35.801891 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:35.801948 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:35.802197 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:35.802200 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:35.802202 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:35.802398 (+ 196us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:35.802507 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:35.803029 (+ 522us) server_negotiation.cc:300] Negotiation successful
0203 21:07:35.803155 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260203 21:07:35.803972 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.793323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:43097 (local address 127.0.102.131:51145)
0203 21:07:35.793758 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:35.793798 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:35.794551 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:35.795048 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:35.795059 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:35.795518 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:35.796052 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.796065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.796998 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.797002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:35.797403 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.797410 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.797607 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.798167 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:35.798191 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:35.799902 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:35.801539 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:35.801544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:35.801556 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:35.801781 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:35.802033 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:35.802035 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:35.802036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:35.802127 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:35.802499 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:35.802506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:35.802786 (+ 280us) client_negotiation.cc:769] Sending connection context
0203 21:07:35.802986 (+ 200us) client_negotiation.cc:240] Negotiation successful
0203 21:07:35.803273 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20260203 21:07:35.804072 8957 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
I20260203 21:07:35.805617 9088 heartbeater.cc:344] Connected to a master server at 127.0.102.190:43097
I20260203 21:07:35.805784 9088 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:35.806384 9088 heartbeater.cc:507] Master 127.0.102.190:43097 requested a full tablet report, sending...
I20260203 21:07:35.807442 8596 ts_manager.cc:194] Registered new tserver with Master: fbfe1cf326d44927abd9f87b4c1daa09 (127.0.102.131:43563)
I20260203 21:07:35.807986 8596 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.102.131@KRBTEST.COM'} at 127.0.102.131:51145
I20260203 21:07:35.812855 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.807236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:46859 (local address 127.0.102.190:43097)
0203 21:07:35.807356 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:35.807360 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:35.807465 (+ 105us) server_negotiation.cc:408] Connection header received
0203 21:07:35.807574 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:35.807578 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:35.807619 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:35.807679 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:35.808241 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.808874 (+ 633us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.809580 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.809702 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.810506 (+ 804us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:35.810521 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:35.810523 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:35.810543 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:35.811691 (+ 1148us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:35.812011 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:35.812013 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:35.812015 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:35.812047 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:35.812266 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:35.812268 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:35.812269 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:35.812383 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:35.812445 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:35.812620 (+ 175us) server_negotiation.cc:300] Negotiation successful
0203 21:07:35.812736 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260203 21:07:35.812873 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.807143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:43097 (local address 127.0.102.131:46859)
0203 21:07:35.807393 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:35.807405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:35.807498 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:35.807703 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:35.807706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:35.807905 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:35.808132 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.808140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.808974 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.808978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:35.809484 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.809493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.809602 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.810087 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:35.810101 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:35.810407 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:35.811789 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:35.811792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:35.811793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:35.811942 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:35.812137 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:35.812141 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:35.812143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:35.812196 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:35.812441 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:35.812443 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:35.812511 (+ 68us) client_negotiation.cc:769] Sending connection context
0203 21:07:35.812585 (+ 74us) client_negotiation.cc:240] Negotiation successful
0203 21:07:35.812717 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
I20260203 21:07:35.813606 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 8957
I20260203 21:07:35.813722 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:07:35.813957 410 external_mini_cluster.cc:1468] Setting key 19428a25e875a66adaffcbdfb206ea99
I20260203 21:07:35.815766 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Feb 03 21:07:35 dist-test-slave-j889 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Feb 03 21:07:35 dist-test-slave-j889 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:35 dist-test-slave-j889 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
W20260203 21:07:35.835124 8648 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:35.830218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39946 (local address 127.0.102.190:43097)
0203 21:07:35.830329 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:35.830332 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:35.830344 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:35.830383 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:35.830386 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:35.830434 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:35.830509 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:35.831238 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.831714 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.832309 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.832425 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.835008 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39946: BlockingRecv error: recv got EOF from 127.0.0.1:39946 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Feb 03 21:07:35 dist-test-slave-j889 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM
I20260203 21:07:35.842450 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.836342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39952 (local address 127.0.102.190:43097)
0203 21:07:35.836448 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:35.836451 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:35.836528 (+ 77us) server_negotiation.cc:408] Connection header received
0203 21:07:35.836612 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:35.836614 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:35.836688 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:35.836750 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:35.837384 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.837848 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.838364 (+ 516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.838493 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.840220 (+ 1727us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:35.840235 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:35.840237 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:35.840259 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:35.841452 (+ 1193us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:35.841727 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:35.841729 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:35.841731 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:35.841764 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:35.841904 (+ 140us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:35.841906 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:35.841907 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:35.842027 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:35.842126 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:35.842260 (+ 134us) server_negotiation.cc:300] Negotiation successful
0203 21:07:35.842338 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260203 21:07:35.844774 8596 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260203 21:07:35.847019 8596 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:07:35.859093 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.855001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:43563 (local address 127.0.0.1:42502)
0203 21:07:35.855709 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:35.855721 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:35.855808 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:35.856364 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:35.856369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:35.856391 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:35.856711 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.856719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.857834 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.857837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:35.858602 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.858611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.858711 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.858764 (+ 53us) client_negotiation.cc:769] Sending connection context
0203 21:07:35.858859 (+ 95us) client_negotiation.cc:240] Negotiation successful
0203 21:07:35.858941 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":99,"threads_started":1}
I20260203 21:07:35.859412 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.855438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:37411 (local address 127.0.0.1:51684)
0203 21:07:35.855727 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:35.855739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:35.855825 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:35.856568 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:35.856571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:35.856587 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:35.856838 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.856846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.858122 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.858126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:35.859096 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.859105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.859199 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.859215 (+ 16us) client_negotiation.cc:769] Sending connection context
0203 21:07:35.859258 (+ 43us) client_negotiation.cc:240] Negotiation successful
0203 21:07:35.859306 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"spinlock_wait_cycles":23296,"thread_start_us":83,"threads_started":1}
I20260203 21:07:35.859579 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.854804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:35873 (local address 127.0.0.1:41740)
0203 21:07:35.855375 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:35.855414 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:35.855548 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:35.857039 (+ 1491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:35.857042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:35.857060 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:35.857275 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.857281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.858440 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.858443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:35.859238 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.859246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.859340 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.859356 (+ 16us) client_negotiation.cc:769] Sending connection context
0203 21:07:35.859401 (+ 45us) client_negotiation.cc:240] Negotiation successful
0203 21:07:35.859456 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":69,"threads_started":1}
I20260203 21:07:35.860142 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.855482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42502 (local address 127.0.102.131:43563)
0203 21:07:35.855857 (+ 375us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:35.855864 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:35.855886 (+ 22us) server_negotiation.cc:408] Connection header received
0203 21:07:35.855971 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:35.855976 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:35.856144 (+ 168us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:35.856280 (+ 136us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:35.856832 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.857734 (+ 902us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.859202 (+ 1468us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.859735 (+ 533us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.859868 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:35.859928 (+ 60us) server_negotiation.cc:300] Negotiation successful
0203 21:07:35.860012 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":97,"threads_started":1}
I20260203 21:07:35.860399 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.855631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.0.102.129:37411)
0203 21:07:35.856010 (+ 379us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:35.856015 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:35.856034 (+ 19us) server_negotiation.cc:408] Connection header received
0203 21:07:35.856114 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:35.856119 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:35.856282 (+ 163us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:35.856426 (+ 144us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:35.856966 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.857985 (+ 1019us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.859564 (+ 1579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.860067 (+ 503us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.860149 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:35.860202 (+ 53us) server_negotiation.cc:300] Negotiation successful
0203 21:07:35.860288 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":76,"threads_started":1}
I20260203 21:07:35.861253 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.854927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.0.102.130:35873)
0203 21:07:35.856590 (+ 1663us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:35.856596 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:35.856615 (+ 19us) server_negotiation.cc:408] Connection header received
0203 21:07:35.856721 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:35.856726 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:35.856881 (+ 155us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:35.857047 (+ 166us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:35.857381 (+ 334us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.858345 (+ 964us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.859853 (+ 1508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.860349 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.860445 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:35.861023 (+ 578us) server_negotiation.cc:300] Negotiation successful
0203 21:07:35.861121 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1558,"thread_start_us":107,"threads_started":1}
I20260203 21:07:35.862882 9022 tablet_service.cc:1505] Processing CreateTablet for tablet b5c52140c37d42c78d9f007c2ef34bb0 (DEFAULT_TABLE table=test-table [id=43e957495392479a819b9bfc5e1d11f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:35.862896 8738 tablet_service.cc:1505] Processing CreateTablet for tablet b5c52140c37d42c78d9f007c2ef34bb0 (DEFAULT_TABLE table=test-table [id=43e957495392479a819b9bfc5e1d11f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:35.863718 9022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5c52140c37d42c78d9f007c2ef34bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:35.863456 8880 tablet_service.cc:1505] Processing CreateTablet for tablet b5c52140c37d42c78d9f007c2ef34bb0 (DEFAULT_TABLE table=test-table [id=43e957495392479a819b9bfc5e1d11f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:35.863744 8738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5c52140c37d42c78d9f007c2ef34bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:35.864177 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5c52140c37d42c78d9f007c2ef34bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:35.869149 9116 tablet_bootstrap.cc:492] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09: Bootstrap starting.
I20260203 21:07:35.870186 9118 tablet_bootstrap.cc:492] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72: Bootstrap starting.
I20260203 21:07:35.871554 9116 tablet_bootstrap.cc:654] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:35.871826 9118 tablet_bootstrap.cc:654] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:35.872735 9116 log.cc:826] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:35.872805 9118 log.cc:826] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:35.872928 9117 tablet_bootstrap.cc:492] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42: Bootstrap starting.
I20260203 21:07:35.874358 9116 tablet_bootstrap.cc:492] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09: No bootstrap required, opened a new log
I20260203 21:07:35.874428 9117 tablet_bootstrap.cc:654] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:35.874529 9116 ts_tablet_manager.cc:1403] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260203 21:07:35.875147 9117 log.cc:826] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:35.875605 9118 tablet_bootstrap.cc:492] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72: No bootstrap required, opened a new log
I20260203 21:07:35.875766 9118 ts_tablet_manager.cc:1403] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260203 21:07:35.876974 9117 tablet_bootstrap.cc:492] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42: No bootstrap required, opened a new log
I20260203 21:07:35.877187 9117 ts_tablet_manager.cc:1403] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260203 21:07:35.877347 9116 raft_consensus.cc:359] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } }
I20260203 21:07:35.877652 9116 raft_consensus.cc:385] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:35.877748 9116 raft_consensus.cc:740] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbfe1cf326d44927abd9f87b4c1daa09, State: Initialized, Role: FOLLOWER
I20260203 21:07:35.878288 9116 consensus_queue.cc:260] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } }
I20260203 21:07:35.878480 9118 raft_consensus.cc:359] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } }
I20260203 21:07:35.878736 9118 raft_consensus.cc:385] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:35.878824 9118 raft_consensus.cc:740] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e95102e3672465d9df500793a178c72, State: Initialized, Role: FOLLOWER
I20260203 21:07:35.879197 9118 consensus_queue.cc:260] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } }
I20260203 21:07:35.880030 9118 ts_tablet_manager.cc:1434] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260203 21:07:35.880148 9116 ts_tablet_manager.cc:1434] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260203 21:07:35.880426 8946 heartbeater.cc:499] Master 127.0.102.190:43097 was elected leader, sending a full tablet report...
I20260203 21:07:35.880599 9088 heartbeater.cc:499] Master 127.0.102.190:43097 was elected leader, sending a full tablet report...
I20260203 21:07:35.880954 9117 raft_consensus.cc:359] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } }
I20260203 21:07:35.881147 9117 raft_consensus.cc:385] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:35.881201 9117 raft_consensus.cc:740] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcee84ea65d44cebf446d3ab030ed42, State: Initialized, Role: FOLLOWER
I20260203 21:07:35.881563 9117 consensus_queue.cc:260] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } }
I20260203 21:07:35.882339 9117 ts_tablet_manager.cc:1434] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260203 21:07:35.883225 8804 heartbeater.cc:499] Master 127.0.102.190:43097 was elected leader, sending a full tablet report...
I20260203 21:07:35.899395 9123 raft_consensus.cc:493] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:07:35.899597 9123 raft_consensus.cc:515] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } }
I20260203 21:07:35.900683 9123 leader_election.cc:290] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbfe1cf326d44927abd9f87b4c1daa09 (127.0.102.131:43563), cfcee84ea65d44cebf446d3ab030ed42 (127.0.102.129:37411)
I20260203 21:07:35.903539 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.900878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:43563 (local address 127.0.102.130:55185)
0203 21:07:35.901001 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:35.901016 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:35.901114 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:35.901428 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:35.901431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:35.901451 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:35.901720 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.901727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.902621 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.902624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:35.903187 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.903194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.903289 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.903310 (+ 21us) client_negotiation.cc:769] Sending connection context
0203 21:07:35.903357 (+ 47us) client_negotiation.cc:240] Negotiation successful
0203 21:07:35.903414 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260203 21:07:35.903813 9125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.901159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:37411 (local address 127.0.102.130:55435)
0203 21:07:35.901469 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:35.901482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:35.901569 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:35.901781 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:35.901784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:35.901803 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:35.902098 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.902106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.902874 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.902878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:35.903460 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:35.903467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.903566 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.903584 (+ 18us) client_negotiation.cc:769] Sending connection context
0203 21:07:35.903630 (+ 46us) client_negotiation.cc:240] Negotiation successful
0203 21:07:35.903679 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":116,"threads_started":1}
I20260203 21:07:35.904017 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.901000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:55185 (local address 127.0.102.131:43563)
0203 21:07:35.901130 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:35.901135 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:35.901183 (+ 48us) server_negotiation.cc:408] Connection header received
0203 21:07:35.901235 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:35.901237 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:35.901279 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:35.901342 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:35.901866 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.902532 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.903298 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.903763 (+ 465us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.903798 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:35.903858 (+ 60us) server_negotiation.cc:300] Negotiation successful
0203 21:07:35.903910 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260203 21:07:35.904258 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.901268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:55435 (local address 127.0.102.129:37411)
0203 21:07:35.901406 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:35.901411 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:35.901542 (+ 131us) server_negotiation.cc:408] Connection header received
0203 21:07:35.901641 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:35.901644 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:35.901682 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:35.901747 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:35.902205 (+ 458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.902766 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.903567 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.904013 (+ 446us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.904049 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:35.904097 (+ 48us) server_negotiation.cc:300] Negotiation successful
0203 21:07:35.904152 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260203 21:07:35.904533 9042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5c52140c37d42c78d9f007c2ef34bb0" candidate_uuid: "6e95102e3672465d9df500793a178c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" is_pre_election: true
I20260203 21:07:35.904803 9042 raft_consensus.cc:2468] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e95102e3672465d9df500793a178c72 in term 0.
I20260203 21:07:35.904781 8758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5c52140c37d42c78d9f007c2ef34bb0" candidate_uuid: "6e95102e3672465d9df500793a178c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcee84ea65d44cebf446d3ab030ed42" is_pre_election: true
I20260203 21:07:35.904995 8758 raft_consensus.cc:2468] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e95102e3672465d9df500793a178c72 in term 0.
I20260203 21:07:35.905225 8832 leader_election.cc:304] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e95102e3672465d9df500793a178c72, fbfe1cf326d44927abd9f87b4c1daa09; no voters:
I20260203 21:07:35.905432 9123 raft_consensus.cc:2804] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:07:35.905499 9123 raft_consensus.cc:493] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:07:35.905550 9123 raft_consensus.cc:3060] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:35.906474 9123 raft_consensus.cc:515] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } }
I20260203 21:07:35.906770 9123 leader_election.cc:290] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [CANDIDATE]: Term 1 election: Requested vote from peers fbfe1cf326d44927abd9f87b4c1daa09 (127.0.102.131:43563), cfcee84ea65d44cebf446d3ab030ed42 (127.0.102.129:37411)
I20260203 21:07:35.907066 9042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5c52140c37d42c78d9f007c2ef34bb0" candidate_uuid: "6e95102e3672465d9df500793a178c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbfe1cf326d44927abd9f87b4c1daa09"
I20260203 21:07:35.907117 8758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5c52140c37d42c78d9f007c2ef34bb0" candidate_uuid: "6e95102e3672465d9df500793a178c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcee84ea65d44cebf446d3ab030ed42"
I20260203 21:07:35.907171 9042 raft_consensus.cc:3060] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:35.907224 8758 raft_consensus.cc:3060] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:35.908078 8758 raft_consensus.cc:2468] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e95102e3672465d9df500793a178c72 in term 1.
I20260203 21:07:35.908114 9042 raft_consensus.cc:2468] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e95102e3672465d9df500793a178c72 in term 1.
I20260203 21:07:35.908329 8832 leader_election.cc:304] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e95102e3672465d9df500793a178c72, cfcee84ea65d44cebf446d3ab030ed42; no voters:
I20260203 21:07:35.908483 9123 raft_consensus.cc:2804] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:35.908710 9123 raft_consensus.cc:697] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [term 1 LEADER]: Becoming Leader. State: Replica: 6e95102e3672465d9df500793a178c72, State: Running, Role: LEADER
I20260203 21:07:35.908991 9123 consensus_queue.cc:237] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } }
I20260203 21:07:35.911733 8596 catalog_manager.cc:5654] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e95102e3672465d9df500793a178c72 (127.0.102.130). New cstate: current_term: 1 leader_uuid: "6e95102e3672465d9df500793a178c72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } health_report { overall_health: UNKNOWN } } }
I20260203 21:07:35.921342 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:35.918654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41754 (local address 127.0.102.130:35873)
0203 21:07:35.918766 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:35.918769 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:35.918781 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:35.918891 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:35.918893 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:35.918940 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:35.919013 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:07:35.919364 (+ 351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.919846 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:35.920420 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:35.920542 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:35.920584 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:07:35.921024 (+ 440us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:07:35.921118 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:35.921210 (+ 92us) server_negotiation.cc:300] Negotiation successful
0203 21:07:35.921251 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260203 21:07:35.926326 8758 raft_consensus.cc:1275] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Refusing update from remote peer 6e95102e3672465d9df500793a178c72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:35.926339 9042 raft_consensus.cc:1275] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Refusing update from remote peer 6e95102e3672465d9df500793a178c72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:35.926872 9123 consensus_queue.cc:1048] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:35.927045 9126 consensus_queue.cc:1048] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:35.934172 9130 mvcc.cc:204] Tried to move back new op lower bound from 7250546097868148736 to 7250546097805414400. Current Snapshot: MvccSnapshot[applied={T|T < 7250546097868148736}]
I20260203 21:07:35.934238 9131 mvcc.cc:204] Tried to move back new op lower bound from 7250546097868148736 to 7250546097805414400. Current Snapshot: MvccSnapshot[applied={T|T < 7250546097868148736}]
I20260203 21:07:35.942279 8596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:07:35.942430 8596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:07:35.944684 8596 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713: Sending DeleteTablet for 3 replicas of tablet b5c52140c37d42c78d9f007c2ef34bb0
I20260203 21:07:35.945343 8880 tablet_service.cc:1552] Processing DeleteTablet for tablet b5c52140c37d42c78d9f007c2ef34bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:35 UTC) from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:41740
I20260203 21:07:35.945412 9022 tablet_service.cc:1552] Processing DeleteTablet for tablet b5c52140c37d42c78d9f007c2ef34bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:35 UTC) from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:42502
I20260203 21:07:35.945806 9140 tablet_replica.cc:333] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09: stopping tablet replica
I20260203 21:07:35.945950 9139 tablet_replica.cc:333] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72: stopping tablet replica
I20260203 21:07:35.945960 8738 tablet_service.cc:1552] Processing DeleteTablet for tablet b5c52140c37d42c78d9f007c2ef34bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:35 UTC) from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:51684
I20260203 21:07:35.946102 9140 raft_consensus.cc:2243] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:35.946161 9139 raft_consensus.cc:2243] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [term 1 LEADER]: Raft consensus shutting down.
I20260203 21:07:35.946398 9140 raft_consensus.cc:2272] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:35.946492 9139 raft_consensus.cc:2272] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:35.947135 8596 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260203 21:07:35.947546 8596 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:07:35.947814 9140 ts_tablet_manager.cc:1916] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:35.947903 9139 ts_tablet_manager.cc:1916] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:35.950193 9139 ts_tablet_manager.cc:1929] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:07:35.950193 9140 ts_tablet_manager.cc:1929] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:07:35.950277 9140 log.cc:1199] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/wal/wals/b5c52140c37d42c78d9f007c2ef34bb0
I20260203 21:07:35.950438 9139 log.cc:1199] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/wal/wals/b5c52140c37d42c78d9f007c2ef34bb0
I20260203 21:07:35.950666 9140 ts_tablet_manager.cc:1950] T b5c52140c37d42c78d9f007c2ef34bb0 P fbfe1cf326d44927abd9f87b4c1daa09: Deleting consensus metadata
I20260203 21:07:35.950785 9139 ts_tablet_manager.cc:1950] T b5c52140c37d42c78d9f007c2ef34bb0 P 6e95102e3672465d9df500793a178c72: Deleting consensus metadata
I20260203 21:07:35.951733 8584 catalog_manager.cc:4985] TS fbfe1cf326d44927abd9f87b4c1daa09 (127.0.102.131:43563): tablet b5c52140c37d42c78d9f007c2ef34bb0 (table test-table [id=43e957495392479a819b9bfc5e1d11f3]) successfully deleted
I20260203 21:07:35.951733 8583 catalog_manager.cc:4985] TS 6e95102e3672465d9df500793a178c72 (127.0.102.130:35873): tablet b5c52140c37d42c78d9f007c2ef34bb0 (table test-table [id=43e957495392479a819b9bfc5e1d11f3]) successfully deleted
I20260203 21:07:35.952190 9141 tablet_replica.cc:333] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42: stopping tablet replica
I20260203 21:07:35.952452 9141 raft_consensus.cc:2243] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:35.952791 9141 raft_consensus.cc:2272] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:35.953210 8738 tablet_service.cc:1505] Processing CreateTablet for tablet ad4cde40f491466da464c9f666431112 (DEFAULT_TABLE table=test-table [id=0a74a572bb1b4d398493443e169971b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:35.953465 8738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad4cde40f491466da464c9f666431112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:35.953625 8880 tablet_service.cc:1505] Processing CreateTablet for tablet ad4cde40f491466da464c9f666431112 (DEFAULT_TABLE table=test-table [id=0a74a572bb1b4d398493443e169971b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:35.953682 9022 tablet_service.cc:1505] Processing CreateTablet for tablet ad4cde40f491466da464c9f666431112 (DEFAULT_TABLE table=test-table [id=0a74a572bb1b4d398493443e169971b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:35.953856 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad4cde40f491466da464c9f666431112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:35.953862 9022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad4cde40f491466da464c9f666431112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:35.955678 9117 tablet_bootstrap.cc:492] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42: Bootstrap starting.
I20260203 21:07:35.955770 9116 tablet_bootstrap.cc:492] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09: Bootstrap starting.
I20260203 21:07:35.956527 9116 tablet_bootstrap.cc:654] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:35.956658 9117 tablet_bootstrap.cc:654] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:35.957304 9118 tablet_bootstrap.cc:492] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72: Bootstrap starting.
I20260203 21:07:35.957738 9116 tablet_bootstrap.cc:492] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09: No bootstrap required, opened a new log
I20260203 21:07:35.957828 9116 ts_tablet_manager.cc:1403] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:35.958204 9118 tablet_bootstrap.cc:654] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:35.958283 9116 raft_consensus.cc:359] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:35.958387 9116 raft_consensus.cc:385] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:35.958415 9116 raft_consensus.cc:740] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbfe1cf326d44927abd9f87b4c1daa09, State: Initialized, Role: FOLLOWER
I20260203 21:07:35.958521 9116 consensus_queue.cc:260] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:35.958787 9116 ts_tablet_manager.cc:1434] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:35.959270 9141 ts_tablet_manager.cc:1916] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:35.959699 9118 tablet_bootstrap.cc:492] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72: No bootstrap required, opened a new log
I20260203 21:07:35.959775 9118 ts_tablet_manager.cc:1403] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260203 21:07:35.960181 9118 raft_consensus.cc:359] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:35.960269 9118 raft_consensus.cc:385] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:35.960297 9118 raft_consensus.cc:740] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e95102e3672465d9df500793a178c72, State: Initialized, Role: FOLLOWER
I20260203 21:07:35.960412 9118 consensus_queue.cc:260] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:35.960672 9118 ts_tablet_manager.cc:1434] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:35.961601 9141 ts_tablet_manager.cc:1929] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:07:35.961658 9141 log.cc:1199] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/wal/wals/b5c52140c37d42c78d9f007c2ef34bb0
I20260203 21:07:35.961859 9141 ts_tablet_manager.cc:1950] T b5c52140c37d42c78d9f007c2ef34bb0 P cfcee84ea65d44cebf446d3ab030ed42: Deleting consensus metadata
I20260203 21:07:35.962185 9117 tablet_bootstrap.cc:492] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42: No bootstrap required, opened a new log
I20260203 21:07:35.962255 9117 ts_tablet_manager.cc:1403] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260203 21:07:35.962656 9117 raft_consensus.cc:359] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:35.962747 9117 raft_consensus.cc:385] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:35.962775 9117 raft_consensus.cc:740] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcee84ea65d44cebf446d3ab030ed42, State: Initialized, Role: FOLLOWER
I20260203 21:07:35.962890 8584 catalog_manager.cc:4985] TS cfcee84ea65d44cebf446d3ab030ed42 (127.0.102.129:37411): tablet b5c52140c37d42c78d9f007c2ef34bb0 (table test-table [id=43e957495392479a819b9bfc5e1d11f3]) successfully deleted
I20260203 21:07:35.962899 9117 consensus_queue.cc:260] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:35.963133 9117 ts_tablet_manager.cc:1434] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260203 21:07:36.055068 9089 tablet.cc:2378] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260203 21:07:36.081151 8947 tablet.cc:2378] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260203 21:07:36.117137 8805 tablet.cc:2378] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260203 21:07:36.140865 9134 raft_consensus.cc:493] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:07:36.141000 9134 raft_consensus.cc:515] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.141382 9134 leader_election.cc:290] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfcee84ea65d44cebf446d3ab030ed42 (127.0.102.129:37411), fbfe1cf326d44927abd9f87b4c1daa09 (127.0.102.131:43563)
I20260203 21:07:36.141875 8758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad4cde40f491466da464c9f666431112" candidate_uuid: "6e95102e3672465d9df500793a178c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcee84ea65d44cebf446d3ab030ed42" is_pre_election: true
I20260203 21:07:36.141940 9042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad4cde40f491466da464c9f666431112" candidate_uuid: "6e95102e3672465d9df500793a178c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" is_pre_election: true
I20260203 21:07:36.142007 8758 raft_consensus.cc:2468] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e95102e3672465d9df500793a178c72 in term 0.
I20260203 21:07:36.142094 9042 raft_consensus.cc:2468] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e95102e3672465d9df500793a178c72 in term 0.
I20260203 21:07:36.142345 8832 leader_election.cc:304] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e95102e3672465d9df500793a178c72, cfcee84ea65d44cebf446d3ab030ed42; no voters:
I20260203 21:07:36.142501 9134 raft_consensus.cc:2804] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:07:36.142560 9134 raft_consensus.cc:493] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:07:36.142596 9134 raft_consensus.cc:3060] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:36.143474 9134 raft_consensus.cc:515] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.143757 9134 leader_election.cc:290] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [CANDIDATE]: Term 1 election: Requested vote from peers cfcee84ea65d44cebf446d3ab030ed42 (127.0.102.129:37411), fbfe1cf326d44927abd9f87b4c1daa09 (127.0.102.131:43563)
I20260203 21:07:36.144158 9042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad4cde40f491466da464c9f666431112" candidate_uuid: "6e95102e3672465d9df500793a178c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbfe1cf326d44927abd9f87b4c1daa09"
I20260203 21:07:36.144179 8758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad4cde40f491466da464c9f666431112" candidate_uuid: "6e95102e3672465d9df500793a178c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcee84ea65d44cebf446d3ab030ed42"
I20260203 21:07:36.144259 8758 raft_consensus.cc:3060] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:36.144266 9042 raft_consensus.cc:3060] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:36.145069 8758 raft_consensus.cc:2468] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e95102e3672465d9df500793a178c72 in term 1.
I20260203 21:07:36.145165 9042 raft_consensus.cc:2468] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e95102e3672465d9df500793a178c72 in term 1.
I20260203 21:07:36.145361 8832 leader_election.cc:304] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e95102e3672465d9df500793a178c72, cfcee84ea65d44cebf446d3ab030ed42; no voters:
I20260203 21:07:36.145535 9134 raft_consensus.cc:2804] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:36.145617 9134 raft_consensus.cc:697] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [term 1 LEADER]: Becoming Leader. State: Replica: 6e95102e3672465d9df500793a178c72, State: Running, Role: LEADER
I20260203 21:07:36.145742 9134 consensus_queue.cc:237] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.147287 8598 catalog_manager.cc:5654] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e95102e3672465d9df500793a178c72 (127.0.102.130). New cstate: current_term: 1 leader_uuid: "6e95102e3672465d9df500793a178c72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } health_report { overall_health: UNKNOWN } } }
Feb 03 21:07:36 dist-test-slave-j889 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.190@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM
I20260203 21:07:36.216972 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.210331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:43097 (local address 127.0.0.1:39964)
0203 21:07:36.210580 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:36.210592 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:36.210696 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:36.210906 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:36.210908 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:36.211324 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:36.211513 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:36.211519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.212159 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.212161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:36.212521 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:36.212526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.212603 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.213108 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:36.213119 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:36.214442 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:36.215886 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:36.215891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:36.215895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:36.216074 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:36.216233 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:36.216235 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:36.216236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:36.216275 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:36.216520 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:36.216525 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:36.216613 (+ 88us) client_negotiation.cc:769] Sending connection context
0203 21:07:36.216713 (+ 100us) client_negotiation.cc:240] Negotiation successful
0203 21:07:36.216813 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":94,"threads_started":1}
I20260203 21:07:36.216991 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.210401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39964 (local address 127.0.102.190:43097)
0203 21:07:36.210537 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:36.210542 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:36.210660 (+ 118us) server_negotiation.cc:408] Connection header received
0203 21:07:36.210769 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:36.210771 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:36.210811 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:36.210872 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:36.211600 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.212068 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.212592 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.212748 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.214555 (+ 1807us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:36.214569 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:36.214571 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:36.214592 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:36.215809 (+ 1217us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:36.216141 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:36.216144 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:36.216145 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:36.216178 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:36.216327 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:36.216329 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:36.216330 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:36.216456 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:36.216518 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:36.216767 (+ 249us) server_negotiation.cc:300] Negotiation successful
0203 21:07:36.216865 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":56}
I20260203 21:07:36.220392 8598 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:39964:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260203 21:07:36.221072 8598 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:07:36.225517 8738 tablet_service.cc:1505] Processing CreateTablet for tablet 9597d1407b84463781eba413598005ed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9e2dc6a4f8a45f6b1eae7f895520b14]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260203 21:07:36.225618 8880 tablet_service.cc:1505] Processing CreateTablet for tablet 9597d1407b84463781eba413598005ed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9e2dc6a4f8a45f6b1eae7f895520b14]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260203 21:07:36.225749 9022 tablet_service.cc:1505] Processing CreateTablet for tablet 9597d1407b84463781eba413598005ed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9e2dc6a4f8a45f6b1eae7f895520b14]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260203 21:07:36.225912 8738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9597d1407b84463781eba413598005ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:36.225978 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9597d1407b84463781eba413598005ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:36.226033 9022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9597d1407b84463781eba413598005ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:36.228048 9118 tablet_bootstrap.cc:492] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72: Bootstrap starting.
I20260203 21:07:36.228473 9117 tablet_bootstrap.cc:492] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42: Bootstrap starting.
I20260203 21:07:36.228523 9116 tablet_bootstrap.cc:492] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09: Bootstrap starting.
I20260203 21:07:36.228957 9118 tablet_bootstrap.cc:654] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:36.229496 9116 tablet_bootstrap.cc:654] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:36.229738 9117 tablet_bootstrap.cc:654] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:36.230118 9118 tablet_bootstrap.cc:492] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72: No bootstrap required, opened a new log
I20260203 21:07:36.230199 9118 ts_tablet_manager.cc:1403] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260203 21:07:36.230602 9116 tablet_bootstrap.cc:492] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09: No bootstrap required, opened a new log
I20260203 21:07:36.230610 9118 raft_consensus.cc:359] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.230681 9116 ts_tablet_manager.cc:1403] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:36.230715 9118 raft_consensus.cc:385] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:36.230751 9118 raft_consensus.cc:740] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e95102e3672465d9df500793a178c72, State: Initialized, Role: FOLLOWER
I20260203 21:07:36.230768 9117 tablet_bootstrap.cc:492] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42: No bootstrap required, opened a new log
I20260203 21:07:36.230835 9117 ts_tablet_manager.cc:1403] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260203 21:07:36.230866 9118 consensus_queue.cc:260] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.231124 9118 ts_tablet_manager.cc:1434] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:36.231103 9134 tablet_replica.cc:442] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } }
I20260203 21:07:36.231119 9116 raft_consensus.cc:359] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.231209 9134 tablet_replica.cc:445] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:07:36.231216 9116 raft_consensus.cc:385] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:36.231249 9116 raft_consensus.cc:740] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbfe1cf326d44927abd9f87b4c1daa09, State: Initialized, Role: FOLLOWER
I20260203 21:07:36.231241 9117 raft_consensus.cc:359] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.231336 9117 raft_consensus.cc:385] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:36.231365 9117 raft_consensus.cc:740] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcee84ea65d44cebf446d3ab030ed42, State: Initialized, Role: FOLLOWER
I20260203 21:07:36.231365 9116 consensus_queue.cc:260] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.231472 9117 consensus_queue.cc:260] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.231592 9122 tablet_replica.cc:442] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } }
I20260203 21:07:36.231705 9122 tablet_replica.cc:445] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:07:36.231699 9117 ts_tablet_manager.cc:1434] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260203 21:07:36.231716 9124 tablet_replica.cc:442] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } }
I20260203 21:07:36.232156 9124 tablet_replica.cc:445] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:07:36.231598 9116 ts_tablet_manager.cc:1434] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:36.244737 9124 raft_consensus.cc:493] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:07:36.244848 9124 raft_consensus.cc:515] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.245841 9124 leader_election.cc:290] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e95102e3672465d9df500793a178c72 (127.0.102.130:35873), fbfe1cf326d44927abd9f87b4c1daa09 (127.0.102.131:43563)
I20260203 21:07:36.248968 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.246044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:43563 (local address 127.0.102.129:59371)
0203 21:07:36.246463 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:36.246477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:36.246574 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:36.246820 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:36.246822 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:36.246835 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:36.247069 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:36.247075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.247941 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.247945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:36.248528 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:36.248536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.248673 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.248693 (+ 20us) client_negotiation.cc:769] Sending connection context
0203 21:07:36.248746 (+ 53us) client_negotiation.cc:240] Negotiation successful
0203 21:07:36.248816 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":120,"threads_started":1}
I20260203 21:07:36.249152 9156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.246038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:35873 (local address 127.0.102.129:40875)
0203 21:07:36.246383 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:36.246395 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:36.246487 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:36.246748 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:36.246751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:36.246766 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:07:36.247039 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:36.247047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.247852 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.247856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:36.248458 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:36.248465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.248567 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.248589 (+ 22us) client_negotiation.cc:769] Sending connection context
0203 21:07:36.248950 (+ 361us) client_negotiation.cc:240] Negotiation successful
0203 21:07:36.249013 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":140,"threads_started":1}
I20260203 21:07:36.249293 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.246133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:40875 (local address 127.0.102.130:35873)
0203 21:07:36.246250 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:36.246254 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:36.246465 (+ 211us) server_negotiation.cc:408] Connection header received
0203 21:07:36.246609 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:36.246612 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:36.246662 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:36.246719 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:36.247164 (+ 445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.247762 (+ 598us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.248563 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.249029 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.249067 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:36.249121 (+ 54us) server_negotiation.cc:300] Negotiation successful
0203 21:07:36.249191 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260203 21:07:36.249348 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.246130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:59371 (local address 127.0.102.131:43563)
0203 21:07:36.246248 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:36.246252 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:36.246544 (+ 292us) server_negotiation.cc:408] Connection header received
0203 21:07:36.246680 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:36.246683 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:36.246731 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:36.246802 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:07:36.247214 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.247841 (+ 627us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.248657 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.249115 (+ 458us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.249147 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:36.249195 (+ 48us) server_negotiation.cc:300] Negotiation successful
0203 21:07:36.249245 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260203 21:07:36.249764 9042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9597d1407b84463781eba413598005ed" candidate_uuid: "cfcee84ea65d44cebf446d3ab030ed42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" is_pre_election: true
I20260203 21:07:36.249889 9042 raft_consensus.cc:2468] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfcee84ea65d44cebf446d3ab030ed42 in term 0.
I20260203 21:07:36.249903 8900 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9597d1407b84463781eba413598005ed" candidate_uuid: "cfcee84ea65d44cebf446d3ab030ed42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e95102e3672465d9df500793a178c72" is_pre_election: true
I20260203 21:07:36.250087 8900 raft_consensus.cc:2468] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfcee84ea65d44cebf446d3ab030ed42 in term 0.
I20260203 21:07:36.250276 8690 leader_election.cc:304] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfcee84ea65d44cebf446d3ab030ed42, fbfe1cf326d44927abd9f87b4c1daa09; no voters:
I20260203 21:07:36.250502 9124 raft_consensus.cc:2804] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:07:36.250584 9124 raft_consensus.cc:493] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:07:36.250636 9124 raft_consensus.cc:3060] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:36.251580 9124 raft_consensus.cc:515] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.252034 9124 leader_election.cc:290] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [CANDIDATE]: Term 1 election: Requested vote from peers 6e95102e3672465d9df500793a178c72 (127.0.102.130:35873), fbfe1cf326d44927abd9f87b4c1daa09 (127.0.102.131:43563)
I20260203 21:07:36.252333 8900 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9597d1407b84463781eba413598005ed" candidate_uuid: "cfcee84ea65d44cebf446d3ab030ed42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e95102e3672465d9df500793a178c72"
I20260203 21:07:36.252394 9042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9597d1407b84463781eba413598005ed" candidate_uuid: "cfcee84ea65d44cebf446d3ab030ed42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbfe1cf326d44927abd9f87b4c1daa09"
I20260203 21:07:36.252446 8900 raft_consensus.cc:3060] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:36.252511 9042 raft_consensus.cc:3060] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:36.253321 8900 raft_consensus.cc:2468] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcee84ea65d44cebf446d3ab030ed42 in term 1.
I20260203 21:07:36.253329 9042 raft_consensus.cc:2468] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcee84ea65d44cebf446d3ab030ed42 in term 1.
I20260203 21:07:36.253590 8689 leader_election.cc:304] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e95102e3672465d9df500793a178c72, cfcee84ea65d44cebf446d3ab030ed42; no voters:
I20260203 21:07:36.253798 9124 raft_consensus.cc:2804] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:36.254010 9124 raft_consensus.cc:697] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [term 1 LEADER]: Becoming Leader. State: Replica: cfcee84ea65d44cebf446d3ab030ed42, State: Running, Role: LEADER
I20260203 21:07:36.254257 9124 consensus_queue.cc:237] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.255566 9157 tablet_replica.cc:442] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42: TxnStatusTablet state changed. Reason: New leader cfcee84ea65d44cebf446d3ab030ed42. Latest consensus state: current_term: 1 leader_uuid: "cfcee84ea65d44cebf446d3ab030ed42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } }
I20260203 21:07:36.255666 9157 tablet_replica.cc:445] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42: This TxnStatusTablet replica's current role is: LEADER
I20260203 21:07:36.255928 9159 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260203 21:07:36.256059 9159 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260203 21:07:36.256476 8598 catalog_manager.cc:5654] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfcee84ea65d44cebf446d3ab030ed42 (127.0.102.129). New cstate: current_term: 1 leader_uuid: "cfcee84ea65d44cebf446d3ab030ed42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } health_report { overall_health: UNKNOWN } } }
I20260203 21:07:36.271077 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.267888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:37411 (local address 127.0.0.1:51690)
0203 21:07:36.268023 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:36.268038 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:36.268160 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:36.268427 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:36.268430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:36.268438 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0203 21:07:36.268614 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:36.268619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.269355 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.269359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:36.269868 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:36.269878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.270004 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.270052 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0203 21:07:36.270742 (+ 690us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0203 21:07:36.270756 (+ 14us) client_negotiation.cc:769] Sending connection context
0203 21:07:36.270864 (+ 108us) client_negotiation.cc:240] Negotiation successful
0203 21:07:36.270939 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260203 21:07:36.271080 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.267961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.0.102.129:37411)
0203 21:07:36.268072 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:36.268077 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:36.268089 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:36.268285 (+ 196us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:36.268287 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:36.268339 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:36.268412 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:07:36.268732 (+ 320us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.269249 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.270001 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.270203 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.270255 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:07:36.270643 (+ 388us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:07:36.270754 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:36.270882 (+ 128us) server_negotiation.cc:300] Negotiation successful
0203 21:07:36.270934 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260203 21:07:36.274188 9042 raft_consensus.cc:1275] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Refusing update from remote peer cfcee84ea65d44cebf446d3ab030ed42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:36.274240 8900 raft_consensus.cc:1275] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72 [term 1 FOLLOWER]: Refusing update from remote peer cfcee84ea65d44cebf446d3ab030ed42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:36.274641 9157 consensus_queue.cc:1048] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:36.274812 9124 consensus_queue.cc:1048] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:36.277007 9122 tablet_replica.cc:442] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09: TxnStatusTablet state changed. Reason: New leader cfcee84ea65d44cebf446d3ab030ed42. Latest consensus state: current_term: 1 leader_uuid: "cfcee84ea65d44cebf446d3ab030ed42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } }
I20260203 21:07:36.277113 9122 tablet_replica.cc:445] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:07:36.277446 9134 tablet_replica.cc:442] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72: TxnStatusTablet state changed. Reason: New leader cfcee84ea65d44cebf446d3ab030ed42. Latest consensus state: current_term: 1 leader_uuid: "cfcee84ea65d44cebf446d3ab030ed42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } }
I20260203 21:07:36.277549 9134 tablet_replica.cc:445] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:07:36.278587 9157 tablet_replica.cc:442] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cfcee84ea65d44cebf446d3ab030ed42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } }
I20260203 21:07:36.278687 9157 tablet_replica.cc:445] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42: This TxnStatusTablet replica's current role is: LEADER
I20260203 21:07:36.278980 9129 mvcc.cc:204] Tried to move back new op lower bound from 7250546099293732864 to 7250546099219525632. Current Snapshot: MvccSnapshot[applied={T|T < 7250546099293732864}]
I20260203 21:07:36.280341 9157 tablet_replica.cc:442] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cfcee84ea65d44cebf446d3ab030ed42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } }
I20260203 21:07:36.280431 9157 tablet_replica.cc:445] T 9597d1407b84463781eba413598005ed P cfcee84ea65d44cebf446d3ab030ed42: This TxnStatusTablet replica's current role is: LEADER
I20260203 21:07:36.280786 9134 tablet_replica.cc:442] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cfcee84ea65d44cebf446d3ab030ed42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } }
I20260203 21:07:36.280882 9134 tablet_replica.cc:445] T 9597d1407b84463781eba413598005ed P 6e95102e3672465d9df500793a178c72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:07:36.282891 9122 tablet_replica.cc:442] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cfcee84ea65d44cebf446d3ab030ed42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } } }
I20260203 21:07:36.282994 9122 tablet_replica.cc:445] T 9597d1407b84463781eba413598005ed P fbfe1cf326d44927abd9f87b4c1daa09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260203 21:07:36.293747 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.290950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:37411 (local address 127.0.102.130:35125)
0203 21:07:36.291216 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:36.291229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:36.291314 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:36.291538 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:36.291540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:36.291546 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0203 21:07:36.291738 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:36.291742 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.292662 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.292665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:36.293066 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:36.293072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.293166 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.293203 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0203 21:07:36.293538 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0203 21:07:36.293542 (+ 4us) client_negotiation.cc:769] Sending connection context
0203 21:07:36.293583 (+ 41us) client_negotiation.cc:240] Negotiation successful
0203 21:07:36.293636 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
I20260203 21:07:36.293776 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.291039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:35125 (local address 127.0.102.129:37411)
0203 21:07:36.291177 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:36.291181 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:36.291289 (+ 108us) server_negotiation.cc:408] Connection header received
0203 21:07:36.291395 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:36.291399 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:36.291454 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:36.291534 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:07:36.291818 (+ 284us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.292557 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.293154 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.293341 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.293385 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:07:36.293463 (+ 78us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:07:36.293537 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:36.293606 (+ 69us) server_negotiation.cc:300] Negotiation successful
0203 21:07:36.293650 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260203 21:07:36.299597 9042 raft_consensus.cc:1275] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Refusing update from remote peer 6e95102e3672465d9df500793a178c72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:36.299674 8758 raft_consensus.cc:1275] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Refusing update from remote peer 6e95102e3672465d9df500793a178c72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:36.300022 9126 consensus_queue.cc:1048] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:36.300280 9126 consensus_queue.cc:1048] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:36.321199 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.318207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:35873 (local address 127.0.102.129:56387)
0203 21:07:36.318559 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:36.318571 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:36.318655 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:36.318877 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:36.318880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:36.318885 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0203 21:07:36.319181 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:36.319189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.319852 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.319855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:36.320439 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:36.320447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.320540 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.320584 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0203 21:07:36.320930 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0203 21:07:36.320936 (+ 6us) client_negotiation.cc:769] Sending connection context
0203 21:07:36.320993 (+ 57us) client_negotiation.cc:240] Negotiation successful
0203 21:07:36.321058 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":93,"threads_started":1}
I20260203 21:07:36.321178 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.318282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:56387 (local address 127.0.102.130:35873)
0203 21:07:36.318399 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:36.318403 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:36.318632 (+ 229us) server_negotiation.cc:408] Connection header received
0203 21:07:36.318731 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:36.318735 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:36.318786 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:36.318875 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:07:36.319287 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.319759 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.320540 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.320746 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.320788 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:07:36.320852 (+ 64us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:07:36.320920 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:36.321001 (+ 81us) server_negotiation.cc:300] Negotiation successful
0203 21:07:36.321046 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260203 21:07:36.339978 8598 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:07:36.340111 8598 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:07:36.341899 8598 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713: Sending DeleteTablet for 3 replicas of tablet ad4cde40f491466da464c9f666431112
I20260203 21:07:36.342402 8880 tablet_service.cc:1552] Processing DeleteTablet for tablet ad4cde40f491466da464c9f666431112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:41740
I20260203 21:07:36.342511 9022 tablet_service.cc:1552] Processing DeleteTablet for tablet ad4cde40f491466da464c9f666431112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:42502
I20260203 21:07:36.342561 9139 tablet_replica.cc:333] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72: stopping tablet replica
I20260203 21:07:36.342664 9139 raft_consensus.cc:2243] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [term 1 LEADER]: Raft consensus shutting down.
I20260203 21:07:36.342666 8738 tablet_service.cc:1552] Processing DeleteTablet for tablet ad4cde40f491466da464c9f666431112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:51684
I20260203 21:07:36.342690 9140 tablet_replica.cc:333] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09: stopping tablet replica
I20260203 21:07:36.342784 9140 raft_consensus.cc:2243] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:36.342834 9139 raft_consensus.cc:2272] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:36.342901 9140 raft_consensus.cc:2272] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:36.343060 9141 tablet_replica.cc:333] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42: stopping tablet replica
I20260203 21:07:36.343164 9141 raft_consensus.cc:2243] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:36.343253 9141 raft_consensus.cc:2272] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:36.343603 9140 ts_tablet_manager.cc:1916] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:36.343874 9141 ts_tablet_manager.cc:1916] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:36.344063 9139 ts_tablet_manager.cc:1916] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:36.346117 9141 ts_tablet_manager.cc:1929] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260203 21:07:36.346118 9140 ts_tablet_manager.cc:1929] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260203 21:07:36.346180 9141 log.cc:1199] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/wal/wals/ad4cde40f491466da464c9f666431112
I20260203 21:07:36.346181 9140 log.cc:1199] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/wal/wals/ad4cde40f491466da464c9f666431112
I20260203 21:07:36.346410 9140 ts_tablet_manager.cc:1950] T ad4cde40f491466da464c9f666431112 P fbfe1cf326d44927abd9f87b4c1daa09: Deleting consensus metadata
I20260203 21:07:36.346410 9141 ts_tablet_manager.cc:1950] T ad4cde40f491466da464c9f666431112 P cfcee84ea65d44cebf446d3ab030ed42: Deleting consensus metadata
I20260203 21:07:36.346571 9139 ts_tablet_manager.cc:1929] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260203 21:07:36.346633 9139 log.cc:1199] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/wal/wals/ad4cde40f491466da464c9f666431112
I20260203 21:07:36.346868 9139 ts_tablet_manager.cc:1950] T ad4cde40f491466da464c9f666431112 P 6e95102e3672465d9df500793a178c72: Deleting consensus metadata
I20260203 21:07:36.347112 8584 catalog_manager.cc:4985] TS cfcee84ea65d44cebf446d3ab030ed42 (127.0.102.129:37411): tablet ad4cde40f491466da464c9f666431112 (table test-table [id=0a74a572bb1b4d398493443e169971b7]) successfully deleted
I20260203 21:07:36.347266 8584 catalog_manager.cc:4985] TS fbfe1cf326d44927abd9f87b4c1daa09 (127.0.102.131:43563): tablet ad4cde40f491466da464c9f666431112 (table test-table [id=0a74a572bb1b4d398493443e169971b7]) successfully deleted
I20260203 21:07:36.347452 8583 catalog_manager.cc:4985] TS 6e95102e3672465d9df500793a178c72 (127.0.102.130:35873): tablet ad4cde40f491466da464c9f666431112 (table test-table [id=0a74a572bb1b4d398493443e169971b7]) successfully deleted
Feb 03 21:07:36 dist-test-slave-j889 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Feb 03 21:07:36 dist-test-slave-j889 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:36 dist-test-slave-j889 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
W20260203 21:07:36.369866 8648 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:36.364315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39978 (local address 127.0.102.190:43097)
0203 21:07:36.364431 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:36.364436 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:36.364620 (+ 184us) server_negotiation.cc:408] Connection header received
0203 21:07:36.364786 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:36.364789 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:36.364844 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:36.364915 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:36.365761 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.366261 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.366892 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.367075 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.369772 (+ 2697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39978: BlockingRecv error: recv got EOF from 127.0.0.1:39978 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Feb 03 21:07:36 dist-test-slave-j889 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM
I20260203 21:07:36.382663 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.374366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.0.102.190:43097)
0203 21:07:36.374482 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:36.374486 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:36.374598 (+ 112us) server_negotiation.cc:408] Connection header received
0203 21:07:36.374712 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:36.374715 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:36.374763 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:36.374847 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:36.375621 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.376382 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.377225 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.377421 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.379661 (+ 2240us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:36.379679 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:36.379683 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:36.379725 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:36.381236 (+ 1511us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:36.381650 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:36.381654 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:36.381656 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:36.381700 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:36.381897 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:36.381900 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:36.381902 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:36.382059 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:36.382172 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:36.382364 (+ 192us) server_negotiation.cc:300] Negotiation successful
0203 21:07:36.382464 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260203 21:07:36.385288 8598 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260203 21:07:36.385725 8598 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:07:36.390267 8738 tablet_service.cc:1505] Processing CreateTablet for tablet 936c7b7f127d4cedad66a52ed3683cc1 (DEFAULT_TABLE table=test-table [id=a469496007fe4671a22368400f16da7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:36.390532 8738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 936c7b7f127d4cedad66a52ed3683cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:36.390596 8880 tablet_service.cc:1505] Processing CreateTablet for tablet 936c7b7f127d4cedad66a52ed3683cc1 (DEFAULT_TABLE table=test-table [id=a469496007fe4671a22368400f16da7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:36.390888 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 936c7b7f127d4cedad66a52ed3683cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:36.392230 9117 tablet_bootstrap.cc:492] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42: Bootstrap starting.
I20260203 21:07:36.392985 9118 tablet_bootstrap.cc:492] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72: Bootstrap starting.
I20260203 21:07:36.393121 9117 tablet_bootstrap.cc:654] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:36.393759 9118 tablet_bootstrap.cc:654] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:36.394006 9022 tablet_service.cc:1505] Processing CreateTablet for tablet 936c7b7f127d4cedad66a52ed3683cc1 (DEFAULT_TABLE table=test-table [id=a469496007fe4671a22368400f16da7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:36.394253 9022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 936c7b7f127d4cedad66a52ed3683cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:36.394301 9117 tablet_bootstrap.cc:492] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42: No bootstrap required, opened a new log
I20260203 21:07:36.394375 9117 ts_tablet_manager.cc:1403] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260203 21:07:36.394788 9118 tablet_bootstrap.cc:492] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72: No bootstrap required, opened a new log
I20260203 21:07:36.394765 9117 raft_consensus.cc:359] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.394860 9117 raft_consensus.cc:385] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:36.394868 9118 ts_tablet_manager.cc:1403] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260203 21:07:36.394891 9117 raft_consensus.cc:740] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcee84ea65d44cebf446d3ab030ed42, State: Initialized, Role: FOLLOWER
I20260203 21:07:36.395007 9117 consensus_queue.cc:260] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.395285 9117 ts_tablet_manager.cc:1434] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260203 21:07:36.395244 9118 raft_consensus.cc:359] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.395335 9118 raft_consensus.cc:385] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:36.395364 9118 raft_consensus.cc:740] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e95102e3672465d9df500793a178c72, State: Initialized, Role: FOLLOWER
I20260203 21:07:36.395475 9118 consensus_queue.cc:260] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.395704 9118 ts_tablet_manager.cc:1434] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:36.396377 9116 tablet_bootstrap.cc:492] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09: Bootstrap starting.
I20260203 21:07:36.397313 9116 tablet_bootstrap.cc:654] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:36.398294 9116 tablet_bootstrap.cc:492] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09: No bootstrap required, opened a new log
I20260203 21:07:36.398370 9116 ts_tablet_manager.cc:1403] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260203 21:07:36.398713 9116 raft_consensus.cc:359] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.398794 9116 raft_consensus.cc:385] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:36.398823 9116 raft_consensus.cc:740] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbfe1cf326d44927abd9f87b4c1daa09, State: Initialized, Role: FOLLOWER
I20260203 21:07:36.398943 9116 consensus_queue.cc:260] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.399174 9116 ts_tablet_manager.cc:1434] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:36.475237 9126 raft_consensus.cc:493] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:07:36.475361 9126 raft_consensus.cc:515] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.475739 9126 leader_election.cc:290] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfcee84ea65d44cebf446d3ab030ed42 (127.0.102.129:37411), fbfe1cf326d44927abd9f87b4c1daa09 (127.0.102.131:43563)
I20260203 21:07:36.476162 8758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "936c7b7f127d4cedad66a52ed3683cc1" candidate_uuid: "6e95102e3672465d9df500793a178c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcee84ea65d44cebf446d3ab030ed42" is_pre_election: true
I20260203 21:07:36.476205 9042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "936c7b7f127d4cedad66a52ed3683cc1" candidate_uuid: "6e95102e3672465d9df500793a178c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" is_pre_election: true
I20260203 21:07:36.476325 9042 raft_consensus.cc:2468] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e95102e3672465d9df500793a178c72 in term 0.
I20260203 21:07:36.476325 8758 raft_consensus.cc:2468] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e95102e3672465d9df500793a178c72 in term 0.
I20260203 21:07:36.476591 8832 leader_election.cc:304] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e95102e3672465d9df500793a178c72, fbfe1cf326d44927abd9f87b4c1daa09; no voters:
I20260203 21:07:36.476766 9126 raft_consensus.cc:2804] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:07:36.476825 9126 raft_consensus.cc:493] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:07:36.476856 9126 raft_consensus.cc:3060] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:36.477743 9126 raft_consensus.cc:515] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.478041 9126 leader_election.cc:290] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [CANDIDATE]: Term 1 election: Requested vote from peers cfcee84ea65d44cebf446d3ab030ed42 (127.0.102.129:37411), fbfe1cf326d44927abd9f87b4c1daa09 (127.0.102.131:43563)
I20260203 21:07:36.478363 8758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "936c7b7f127d4cedad66a52ed3683cc1" candidate_uuid: "6e95102e3672465d9df500793a178c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcee84ea65d44cebf446d3ab030ed42"
I20260203 21:07:36.478386 9042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "936c7b7f127d4cedad66a52ed3683cc1" candidate_uuid: "6e95102e3672465d9df500793a178c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbfe1cf326d44927abd9f87b4c1daa09"
I20260203 21:07:36.478487 9042 raft_consensus.cc:3060] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:36.478487 8758 raft_consensus.cc:3060] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:36.479517 8758 raft_consensus.cc:2468] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e95102e3672465d9df500793a178c72 in term 1.
I20260203 21:07:36.479517 9042 raft_consensus.cc:2468] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e95102e3672465d9df500793a178c72 in term 1.
I20260203 21:07:36.479826 8832 leader_election.cc:304] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e95102e3672465d9df500793a178c72, fbfe1cf326d44927abd9f87b4c1daa09; no voters:
I20260203 21:07:36.480036 9126 raft_consensus.cc:2804] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:36.480113 9126 raft_consensus.cc:697] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [term 1 LEADER]: Becoming Leader. State: Replica: 6e95102e3672465d9df500793a178c72, State: Running, Role: LEADER
I20260203 21:07:36.480248 9126 consensus_queue.cc:237] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.481899 8596 catalog_manager.cc:5654] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e95102e3672465d9df500793a178c72 (127.0.102.130). New cstate: current_term: 1 leader_uuid: "6e95102e3672465d9df500793a178c72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } health_report { overall_health: UNKNOWN } } }
I20260203 21:07:36.498687 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.496283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41766 (local address 127.0.102.130:35873)
0203 21:07:36.496437 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:36.496441 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:36.496457 (+ 16us) server_negotiation.cc:408] Connection header received
0203 21:07:36.496501 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:36.496504 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:36.496551 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:36.496618 (+ 67us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:07:36.497074 (+ 456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.497535 (+ 461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.498167 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.498291 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.498325 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:07:36.498389 (+ 64us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:07:36.498440 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:36.498509 (+ 69us) server_negotiation.cc:300] Negotiation successful
0203 21:07:36.498550 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260203 21:07:36.500586 9042 raft_consensus.cc:1275] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Refusing update from remote peer 6e95102e3672465d9df500793a178c72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:36.500653 8758 raft_consensus.cc:1275] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Refusing update from remote peer 6e95102e3672465d9df500793a178c72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:36.501003 9126 consensus_queue.cc:1048] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:36.501144 9123 consensus_queue.cc:1048] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:36.507869 8599 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:07:36.507987 8599 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:07:36.509672 8599 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713: Sending DeleteTablet for 3 replicas of tablet 936c7b7f127d4cedad66a52ed3683cc1
I20260203 21:07:36.510118 8738 tablet_service.cc:1552] Processing DeleteTablet for tablet 936c7b7f127d4cedad66a52ed3683cc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:51684
I20260203 21:07:36.510128 8880 tablet_service.cc:1552] Processing DeleteTablet for tablet 936c7b7f127d4cedad66a52ed3683cc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:41740
I20260203 21:07:36.510154 9022 tablet_service.cc:1552] Processing DeleteTablet for tablet 936c7b7f127d4cedad66a52ed3683cc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:42502
I20260203 21:07:36.510278 9139 tablet_replica.cc:333] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72: stopping tablet replica
I20260203 21:07:36.510284 9141 tablet_replica.cc:333] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42: stopping tablet replica
I20260203 21:07:36.510365 9141 raft_consensus.cc:2243] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:36.510375 9139 raft_consensus.cc:2243] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [term 1 LEADER]: Raft consensus shutting down.
I20260203 21:07:36.510452 9141 raft_consensus.cc:2272] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:36.510540 9139 raft_consensus.cc:2272] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:36.510589 9140 tablet_replica.cc:333] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09: stopping tablet replica
I20260203 21:07:36.510680 9140 raft_consensus.cc:2243] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:36.510783 9140 raft_consensus.cc:2272] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:36.510845 8599 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260203 21:07:36.511041 9141 ts_tablet_manager.cc:1916] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:36.511127 9139 ts_tablet_manager.cc:1916] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260203 21:07:36.511229 8599 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:07:36.511656 9140 ts_tablet_manager.cc:1916] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:36.513175 9139 ts_tablet_manager.cc:1929] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:07:36.513175 9141 ts_tablet_manager.cc:1929] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:07:36.513247 9139 log.cc:1199] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/wal/wals/936c7b7f127d4cedad66a52ed3683cc1
I20260203 21:07:36.513247 9141 log.cc:1199] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/wal/wals/936c7b7f127d4cedad66a52ed3683cc1
I20260203 21:07:36.513471 9141 ts_tablet_manager.cc:1950] T 936c7b7f127d4cedad66a52ed3683cc1 P cfcee84ea65d44cebf446d3ab030ed42: Deleting consensus metadata
I20260203 21:07:36.513486 9139 ts_tablet_manager.cc:1950] T 936c7b7f127d4cedad66a52ed3683cc1 P 6e95102e3672465d9df500793a178c72: Deleting consensus metadata
I20260203 21:07:36.513736 9140 ts_tablet_manager.cc:1929] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:07:36.513794 9140 log.cc:1199] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/wal/wals/936c7b7f127d4cedad66a52ed3683cc1
I20260203 21:07:36.514019 9140 ts_tablet_manager.cc:1950] T 936c7b7f127d4cedad66a52ed3683cc1 P fbfe1cf326d44927abd9f87b4c1daa09: Deleting consensus metadata
I20260203 21:07:36.514035 8583 catalog_manager.cc:4985] TS 6e95102e3672465d9df500793a178c72 (127.0.102.130:35873): tablet 936c7b7f127d4cedad66a52ed3683cc1 (table test-table [id=a469496007fe4671a22368400f16da7d]) successfully deleted
I20260203 21:07:36.514550 8584 catalog_manager.cc:4985] TS fbfe1cf326d44927abd9f87b4c1daa09 (127.0.102.131:43563): tablet 936c7b7f127d4cedad66a52ed3683cc1 (table test-table [id=a469496007fe4671a22368400f16da7d]) successfully deleted
I20260203 21:07:36.514675 8584 catalog_manager.cc:4985] TS cfcee84ea65d44cebf446d3ab030ed42 (127.0.102.129:37411): tablet 936c7b7f127d4cedad66a52ed3683cc1 (table test-table [id=a469496007fe4671a22368400f16da7d]) successfully deleted
I20260203 21:07:36.515759 8738 tablet_service.cc:1505] Processing CreateTablet for tablet c3d1b21b032b4b9ab0c2a9e973901b51 (DEFAULT_TABLE table=test-table [id=b93f3f04c9e84657aa8e5bf9cbaf52df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:36.515993 8738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d1b21b032b4b9ab0c2a9e973901b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:36.515959 8880 tablet_service.cc:1505] Processing CreateTablet for tablet c3d1b21b032b4b9ab0c2a9e973901b51 (DEFAULT_TABLE table=test-table [id=b93f3f04c9e84657aa8e5bf9cbaf52df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:36.516163 8880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d1b21b032b4b9ab0c2a9e973901b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:36.516189 9022 tablet_service.cc:1505] Processing CreateTablet for tablet c3d1b21b032b4b9ab0c2a9e973901b51 (DEFAULT_TABLE table=test-table [id=b93f3f04c9e84657aa8e5bf9cbaf52df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:07:36.516397 9022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d1b21b032b4b9ab0c2a9e973901b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:36.517702 9118 tablet_bootstrap.cc:492] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72: Bootstrap starting.
I20260203 21:07:36.517938 9117 tablet_bootstrap.cc:492] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42: Bootstrap starting.
I20260203 21:07:36.518214 9116 tablet_bootstrap.cc:492] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09: Bootstrap starting.
I20260203 21:07:36.518525 9118 tablet_bootstrap.cc:654] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:36.518651 9117 tablet_bootstrap.cc:654] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:36.519100 9116 tablet_bootstrap.cc:654] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:36.519520 9118 tablet_bootstrap.cc:492] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72: No bootstrap required, opened a new log
I20260203 21:07:36.519598 9118 ts_tablet_manager.cc:1403] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:36.519680 9117 tablet_bootstrap.cc:492] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42: No bootstrap required, opened a new log
I20260203 21:07:36.519752 9117 ts_tablet_manager.cc:1403] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260203 21:07:36.519969 9118 raft_consensus.cc:359] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.520042 9118 raft_consensus.cc:385] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:36.520061 9118 raft_consensus.cc:740] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e95102e3672465d9df500793a178c72, State: Initialized, Role: FOLLOWER
I20260203 21:07:36.520154 9116 tablet_bootstrap.cc:492] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09: No bootstrap required, opened a new log
I20260203 21:07:36.520160 9118 consensus_queue.cc:260] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.520145 9117 raft_consensus.cc:359] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.520217 9116 ts_tablet_manager.cc:1403] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260203 21:07:36.520247 9117 raft_consensus.cc:385] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:36.520275 9117 raft_consensus.cc:740] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcee84ea65d44cebf446d3ab030ed42, State: Initialized, Role: FOLLOWER
I20260203 21:07:36.520416 9118 ts_tablet_manager.cc:1434] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:36.520385 9117 consensus_queue.cc:260] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.520612 9117 ts_tablet_manager.cc:1434] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260203 21:07:36.520602 9116 raft_consensus.cc:359] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.520727 9116 raft_consensus.cc:385] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:36.520758 9116 raft_consensus.cc:740] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbfe1cf326d44927abd9f87b4c1daa09, State: Initialized, Role: FOLLOWER
I20260203 21:07:36.520871 9116 consensus_queue.cc:260] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.521134 9116 ts_tablet_manager.cc:1434] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260203 21:07:36.794330 9164 raft_consensus.cc:493] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:07:36.794472 9164 raft_consensus.cc:515] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.794898 9164 leader_election.cc:290] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e95102e3672465d9df500793a178c72 (127.0.102.130:35873), fbfe1cf326d44927abd9f87b4c1daa09 (127.0.102.131:43563)
I20260203 21:07:36.795311 8900 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d1b21b032b4b9ab0c2a9e973901b51" candidate_uuid: "cfcee84ea65d44cebf446d3ab030ed42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e95102e3672465d9df500793a178c72" is_pre_election: true
I20260203 21:07:36.795396 9042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d1b21b032b4b9ab0c2a9e973901b51" candidate_uuid: "cfcee84ea65d44cebf446d3ab030ed42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" is_pre_election: true
I20260203 21:07:36.795460 8900 raft_consensus.cc:2468] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfcee84ea65d44cebf446d3ab030ed42 in term 0.
I20260203 21:07:36.795516 9042 raft_consensus.cc:2468] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfcee84ea65d44cebf446d3ab030ed42 in term 0.
I20260203 21:07:36.795744 8689 leader_election.cc:304] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e95102e3672465d9df500793a178c72, cfcee84ea65d44cebf446d3ab030ed42; no voters:
I20260203 21:07:36.795915 9164 raft_consensus.cc:2804] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:07:36.796010 9164 raft_consensus.cc:493] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:07:36.796066 9164 raft_consensus.cc:3060] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:36.797148 9164 raft_consensus.cc:515] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.797475 9164 leader_election.cc:290] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [CANDIDATE]: Term 1 election: Requested vote from peers 6e95102e3672465d9df500793a178c72 (127.0.102.130:35873), fbfe1cf326d44927abd9f87b4c1daa09 (127.0.102.131:43563)
I20260203 21:07:36.797772 8900 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d1b21b032b4b9ab0c2a9e973901b51" candidate_uuid: "cfcee84ea65d44cebf446d3ab030ed42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e95102e3672465d9df500793a178c72"
I20260203 21:07:36.797796 9042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3d1b21b032b4b9ab0c2a9e973901b51" candidate_uuid: "cfcee84ea65d44cebf446d3ab030ed42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbfe1cf326d44927abd9f87b4c1daa09"
I20260203 21:07:36.797887 8900 raft_consensus.cc:3060] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:36.797887 9042 raft_consensus.cc:3060] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:36.798835 9042 raft_consensus.cc:2468] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcee84ea65d44cebf446d3ab030ed42 in term 1.
I20260203 21:07:36.798851 8900 raft_consensus.cc:2468] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfcee84ea65d44cebf446d3ab030ed42 in term 1.
I20260203 21:07:36.799149 8689 leader_election.cc:304] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e95102e3672465d9df500793a178c72, cfcee84ea65d44cebf446d3ab030ed42; no voters:
I20260203 21:07:36.799347 9164 raft_consensus.cc:2804] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:36.799441 9164 raft_consensus.cc:697] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 LEADER]: Becoming Leader. State: Replica: cfcee84ea65d44cebf446d3ab030ed42, State: Running, Role: LEADER
I20260203 21:07:36.799587 9164 consensus_queue.cc:237] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } }
I20260203 21:07:36.801215 8598 catalog_manager.cc:5654] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfcee84ea65d44cebf446d3ab030ed42 (127.0.102.129). New cstate: current_term: 1 leader_uuid: "cfcee84ea65d44cebf446d3ab030ed42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfcee84ea65d44cebf446d3ab030ed42" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 } health_report { overall_health: UNKNOWN } } }
I20260203 21:07:36.819728 9196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.816921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.0.102.129:37411)
0203 21:07:36.817134 (+ 213us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:36.817142 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:36.817154 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:36.817188 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:36.817190 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:36.817233 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:36.817302 (+ 69us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:07:36.817685 (+ 383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.818248 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.819093 (+ 845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.819248 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.819287 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:07:36.819353 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:07:36.819411 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:36.819533 (+ 122us) server_negotiation.cc:300] Negotiation successful
0203 21:07:36.819586 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
I20260203 21:07:36.827474 9196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.824780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:46283 (local address 127.0.102.129:37411)
0203 21:07:36.824885 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:36.824889 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:36.825018 (+ 129us) server_negotiation.cc:408] Connection header received
0203 21:07:36.825116 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:36.825120 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:36.825169 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:36.825240 (+ 71us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:07:36.825558 (+ 318us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.826271 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.826922 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.827060 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.827094 (+ 34us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:07:36.827152 (+ 58us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:07:36.827208 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:36.827334 (+ 126us) server_negotiation.cc:300] Negotiation successful
0203 21:07:36.827370 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260203 21:07:36.827489 9199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.824740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:37411 (local address 127.0.102.129:46283)
0203 21:07:36.824952 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:36.824963 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:36.825049 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:36.825280 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:36.825283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:36.825290 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0203 21:07:36.825473 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:36.825478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.826390 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:36.826393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:36.826820 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:36.826828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:36.826935 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:36.826959 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0203 21:07:36.827253 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0203 21:07:36.827259 (+ 6us) client_negotiation.cc:769] Sending connection context
0203 21:07:36.827327 (+ 68us) client_negotiation.cc:240] Negotiation successful
0203 21:07:36.827385 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":50,"threads_started":1}
I20260203 21:07:36.832314 9042 raft_consensus.cc:1275] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Refusing update from remote peer cfcee84ea65d44cebf446d3ab030ed42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:36.832365 8900 raft_consensus.cc:1275] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72 [term 1 FOLLOWER]: Refusing update from remote peer cfcee84ea65d44cebf446d3ab030ed42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:07:36.832732 9124 consensus_queue.cc:1048] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e95102e3672465d9df500793a178c72" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:36.832861 9164 consensus_queue.cc:1048] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbfe1cf326d44927abd9f87b4c1daa09" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:07:36.862536 8596 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:07:36.862700 8596 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:07:36.864593 8596 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 21c8fb97ac064d7cb5c7d4828cfba713: Sending DeleteTablet for 3 replicas of tablet c3d1b21b032b4b9ab0c2a9e973901b51
I20260203 21:07:36.865177 8880 tablet_service.cc:1552] Processing DeleteTablet for tablet c3d1b21b032b4b9ab0c2a9e973901b51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:41740
I20260203 21:07:36.865291 9022 tablet_service.cc:1552] Processing DeleteTablet for tablet c3d1b21b032b4b9ab0c2a9e973901b51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:42502
I20260203 21:07:36.865391 9139 tablet_replica.cc:333] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72: stopping tablet replica
I20260203 21:07:36.865504 9140 tablet_replica.cc:333] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09: stopping tablet replica
I20260203 21:07:36.865500 9139 raft_consensus.cc:2243] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:36.865589 9140 raft_consensus.cc:2243] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:07:36.865604 9139 raft_consensus.cc:2272] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:36.865664 9140 raft_consensus.cc:2272] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:36.866156 8738 tablet_service.cc:1552] Processing DeleteTablet for tablet c3d1b21b032b4b9ab0c2a9e973901b51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:07:36 UTC) from {username='oryx', principal='oryx/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:51684
I20260203 21:07:36.866321 9141 tablet_replica.cc:333] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42: stopping tablet replica
I20260203 21:07:36.866365 9139 ts_tablet_manager.cc:1916] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:36.866418 9141 raft_consensus.cc:2243] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 LEADER]: Raft consensus shutting down.
I20260203 21:07:36.866608 9140 ts_tablet_manager.cc:1916] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:36.866688 9141 raft_consensus.cc:2272] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:07:36.867287 9141 ts_tablet_manager.cc:1916] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:07:36.868777 9139 ts_tablet_manager.cc:1929] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260203 21:07:36.868788 9140 ts_tablet_manager.cc:1929] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260203 21:07:36.868841 9139 log.cc:1199] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-1/wal/wals/c3d1b21b032b4b9ab0c2a9e973901b51
I20260203 21:07:36.868849 9140 log.cc:1199] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-2/wal/wals/c3d1b21b032b4b9ab0c2a9e973901b51
I20260203 21:07:36.869088 9140 ts_tablet_manager.cc:1950] T c3d1b21b032b4b9ab0c2a9e973901b51 P fbfe1cf326d44927abd9f87b4c1daa09: Deleting consensus metadata
I20260203 21:07:36.869104 9139 ts_tablet_manager.cc:1950] T c3d1b21b032b4b9ab0c2a9e973901b51 P 6e95102e3672465d9df500793a178c72: Deleting consensus metadata
I20260203 21:07:36.869451 9141 ts_tablet_manager.cc:1929] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260203 21:07:36.869516 9141 log.cc:1199] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1770152788143875-410-0/minicluster-data/ts-0/wal/wals/c3d1b21b032b4b9ab0c2a9e973901b51
I20260203 21:07:36.869680 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 8673
I20260203 21:07:36.869725 9141 ts_tablet_manager.cc:1950] T c3d1b21b032b4b9ab0c2a9e973901b51 P cfcee84ea65d44cebf446d3ab030ed42: Deleting consensus metadata
I20260203 21:07:36.871186 8584 catalog_manager.cc:4985] TS fbfe1cf326d44927abd9f87b4c1daa09 (127.0.102.131:43563): tablet c3d1b21b032b4b9ab0c2a9e973901b51 (table test-table [id=b93f3f04c9e84657aa8e5bf9cbaf52df]) successfully deleted
I20260203 21:07:36.874224 8583 catalog_manager.cc:4985] TS 6e95102e3672465d9df500793a178c72 (127.0.102.130:35873): tablet c3d1b21b032b4b9ab0c2a9e973901b51 (table test-table [id=b93f3f04c9e84657aa8e5bf9cbaf52df]) successfully deleted
W20260203 21:07:36.876399 8584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.102.129:37411 (error 108)
I20260203 21:07:36.876458 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 8815
I20260203 21:07:36.878427 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:36.878106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:37411 (local address 127.0.0.1:51698)
0203 21:07:36.878298 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:36.878349 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.129:37411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":53,"threads_started":1}
W20260203 21:07:36.878536 8584 catalog_manager.cc:4712] TS cfcee84ea65d44cebf446d3ab030ed42 (127.0.102.129:37411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3d1b21b032b4b9ab0c2a9e973901b51: Network error: Client connection negotiation failed: client connection to 127.0.102.129:37411: connect: Connection refused (error 111)
I20260203 21:07:36.882341 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 8957
I20260203 21:07:36.887296 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 8567
2026-02-03T21:07:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3903 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:07:36 dist-test-slave-j889 krb5kdc[9208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:07:36 dist-test-slave-j889 krb5kdc[9208](info): set up 2 sockets
Feb 03 21:07:36 dist-test-slave-j889 krb5kdc[9208](info): commencing operation
krb5kdc: starting...
W20260203 21:07:38.717305 410 mini_kdc.cc:121] Time spent starting KDC: real 1.809s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:07:38 dist-test-slave-j889 krb5kdc[9208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:07:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:07:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:07:38.863490 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:44353
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:40973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:44353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:38.972302 9224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:38.972568 9224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:38.972671 9224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:38.976853 9224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:07:38.976949 9224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:38.976984 9224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:38.977006 9224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:07:38.977030 9224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:07:38.981871 9224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:40973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:44353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:44353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:38.983067 9224 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:38.983928 9224 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:38.989562 9230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:38.989567 9232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:38.989566 9229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:38.989645 9224 server_base.cc:1060] running on GCE node
I20260203 21:07:38.990290 9224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:38.991214 9224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:38.992362 9224 hybrid_clock.cc:648] HybridClock initialized: now 1770152858992338 us; error 36 us; skew 500 ppm
W20260203 21:07:38.995492 9224 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-02-03T21:07:39Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2114 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:07:39 dist-test-slave-j889 krb5kdc[9237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:07:39 dist-test-slave-j889 krb5kdc[9237](info): set up 2 sockets
Feb 03 21:07:39 dist-test-slave-j889 krb5kdc[9237](info): commencing operation
krb5kdc: starting...
W20260203 21:07:40.857299 410 mini_kdc.cc:121] Time spent starting KDC: real 1.834s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:07:40 dist-test-slave-j889 krb5kdc[9237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:07:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:07:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:07:40.995788 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:46645
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:46645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:41.108210 9253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:41.108484 9253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:41.108573 9253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:41.112056 9253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:07:41.112138 9253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:41.112169 9253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:41.112195 9253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:07:41.112226 9253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:07:41.116742 9253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:46645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:46645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:41.117940 9253 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:41.118835 9253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:41.124858 9261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:41.124859 9259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:41.124861 9258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:41.124958 9253 server_base.cc:1060] running on GCE node
I20260203 21:07:41.125556 9253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:41.126505 9253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:41.127678 9253 hybrid_clock.cc:648] HybridClock initialized: now 1770152861127631 us; error 63 us; skew 500 ppm
Feb 03 21:07:41 dist-test-slave-j889 krb5kdc[9237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:41.130381 9253 init.cc:377] Logged in from keytab as kudu/127.0.102.190@KRBTEST.COM (short username kudu)
I20260203 21:07:41.131575 9253 webserver.cc:492] Webserver started at http://127.0.102.190:42763/ using document root <none> and password file <none>
I20260203 21:07:41.132133 9253 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:41.132189 9253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:41.132370 9253 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:41.134177 9253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "b4a2dd4f57a74e97a84571b7a73818ba"
format_stamp: "Formatted at 2026-02-03 21:07:41 on dist-test-slave-j889"
server_key: "e445d7375fc54422e5dd65b09fbb3813"
server_key_iv: "279b3926a44a3a2b455d593065f2d4f0"
server_key_version: "encryptionkey@0"
I20260203 21:07:41.134657 9253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "b4a2dd4f57a74e97a84571b7a73818ba"
format_stamp: "Formatted at 2026-02-03 21:07:41 on dist-test-slave-j889"
server_key: "e445d7375fc54422e5dd65b09fbb3813"
server_key_iv: "279b3926a44a3a2b455d593065f2d4f0"
server_key_version: "encryptionkey@0"
I20260203 21:07:41.138253 9253 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260203 21:07:41.140879 9268 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:41.142156 9253 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260203 21:07:41.142300 9253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "b4a2dd4f57a74e97a84571b7a73818ba"
format_stamp: "Formatted at 2026-02-03 21:07:41 on dist-test-slave-j889"
server_key: "e445d7375fc54422e5dd65b09fbb3813"
server_key_iv: "279b3926a44a3a2b455d593065f2d4f0"
server_key_version: "encryptionkey@0"
I20260203 21:07:41.142408 9253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:41.170862 9253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:41.173825 9253 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:41.173998 9253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:41.181190 9253 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:46645
I20260203 21:07:41.181185 9320 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:46645 every 8 connection(s)
I20260203 21:07:41.182246 9253 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:07:41.185304 9321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:41.190992 9321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba: Bootstrap starting.
I20260203 21:07:41.192168 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 9253
I20260203 21:07:41.192297 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:07:41.192567 410 external_mini_cluster.cc:1468] Setting key ce6ffd1d75ef6e08cff74f9ab5911239
I20260203 21:07:41.193714 9321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:41.194516 9321 log.cc:826] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:41.196411 9321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba: No bootstrap required, opened a new log
I20260203 21:07:41.198998 9321 raft_consensus.cc:359] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a2dd4f57a74e97a84571b7a73818ba" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46645 } }
I20260203 21:07:41.199194 9321 raft_consensus.cc:385] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:41.199250 9321 raft_consensus.cc:740] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a2dd4f57a74e97a84571b7a73818ba, State: Initialized, Role: FOLLOWER
I20260203 21:07:41.199769 9321 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a2dd4f57a74e97a84571b7a73818ba" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46645 } }
Feb 03 21:07:41 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:07:41.199923 9321 raft_consensus.cc:399] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:07:41.199990 9321 raft_consensus.cc:493] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:07:41.200078 9321 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:41.201123 9321 raft_consensus.cc:515] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a2dd4f57a74e97a84571b7a73818ba" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46645 } }
I20260203 21:07:41.201494 9321 leader_election.cc:304] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4a2dd4f57a74e97a84571b7a73818ba; no voters:
I20260203 21:07:41.201812 9321 leader_election.cc:290] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:07:41.201937 9326 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:41.202147 9326 raft_consensus.cc:697] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 1 LEADER]: Becoming Leader. State: Replica: b4a2dd4f57a74e97a84571b7a73818ba, State: Running, Role: LEADER
I20260203 21:07:41.202535 9326 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a2dd4f57a74e97a84571b7a73818ba" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46645 } }
I20260203 21:07:41.202880 9321 sys_catalog.cc:565] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:07:41.203822 9327 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4a2dd4f57a74e97a84571b7a73818ba. Latest consensus state: current_term: 1 leader_uuid: "b4a2dd4f57a74e97a84571b7a73818ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a2dd4f57a74e97a84571b7a73818ba" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46645 } } }
I20260203 21:07:41.203927 9327 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:41.204535 9328 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4a2dd4f57a74e97a84571b7a73818ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a2dd4f57a74e97a84571b7a73818ba" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46645 } } }
I20260203 21:07:41.204715 9328 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:41.206736 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:41.193756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.0.102.190:46645)
0203 21:07:41.194209 (+ 453us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:41.194220 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:41.194254 (+ 34us) server_negotiation.cc:408] Connection header received
0203 21:07:41.194839 (+ 585us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:41.194885 (+ 46us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:41.195267 (+ 382us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:41.195616 (+ 349us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:41.196508 (+ 892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:41.197297 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:41.197978 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:41.198263 (+ 285us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:41.200554 (+ 2291us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:41.200576 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:41.200587 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:41.200666 (+ 79us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:41.203301 (+ 2635us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:41.203841 (+ 540us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:41.203847 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:41.203853 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:41.203958 (+ 105us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:41.204228 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:41.204232 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:41.204234 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:41.204741 (+ 507us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:41.204956 (+ 215us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:41.205827 (+ 871us) server_negotiation.cc:300] Negotiation successful
0203 21:07:41.206034 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":148,"threads_started":1}
W20260203 21:07:41.207449 9341 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260203 21:07:41.207511 9341 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260203 21:07:41.207579 9335 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:07:41.208293 9335 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:07:41.213546 9335 catalog_manager.cc:1357] Generated new cluster ID: 7877b683d5904de7a324c1f35a566300
I20260203 21:07:41.213634 9335 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:07:41.231617 9335 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:07:41.232614 9335 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:07:41.245783 9335 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba: Generated new TSK 0
I20260203 21:07:41.246573 9335 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:07:41.308604 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:46645
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:41.434475 9349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:41.434736 9349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:41.434814 9349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:41.438226 9349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:41.438308 9349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:41.438412 9349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:07:41.442920 9349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:46645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:41.444084 9349 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:41.444993 9349 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:41.451962 9355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:41.452131 9357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:41.452189 9354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:41.452028 9349 server_base.cc:1060] running on GCE node
I20260203 21:07:41.452591 9349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:41.453202 9349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:41.454371 9349 hybrid_clock.cc:648] HybridClock initialized: now 1770152861454340 us; error 46 us; skew 500 ppm
Feb 03 21:07:41 dist-test-slave-j889 krb5kdc[9237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:41.457302 9349 init.cc:377] Logged in from keytab as kudu/127.0.102.129@KRBTEST.COM (short username kudu)
I20260203 21:07:41.458473 9349 webserver.cc:492] Webserver started at http://127.0.102.129:39615/ using document root <none> and password file <none>
I20260203 21:07:41.459051 9349 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:41.459110 9349 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:41.459302 9349 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:41.461159 9349 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "e7d63ca3af004e71a32911a73cfd3457"
format_stamp: "Formatted at 2026-02-03 21:07:41 on dist-test-slave-j889"
server_key: "3f9a0094b26bfe62a37a8ea697c99df9"
server_key_iv: "76eb665684fd1f8837521aba47e5d3bc"
server_key_version: "encryptionkey@0"
I20260203 21:07:41.461652 9349 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "e7d63ca3af004e71a32911a73cfd3457"
format_stamp: "Formatted at 2026-02-03 21:07:41 on dist-test-slave-j889"
server_key: "3f9a0094b26bfe62a37a8ea697c99df9"
server_key_iv: "76eb665684fd1f8837521aba47e5d3bc"
server_key_version: "encryptionkey@0"
I20260203 21:07:41.465571 9349 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260203 21:07:41.468261 9364 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:41.469527 9349 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:07:41.469681 9349 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "e7d63ca3af004e71a32911a73cfd3457"
format_stamp: "Formatted at 2026-02-03 21:07:41 on dist-test-slave-j889"
server_key: "3f9a0094b26bfe62a37a8ea697c99df9"
server_key_iv: "76eb665684fd1f8837521aba47e5d3bc"
server_key_version: "encryptionkey@0"
I20260203 21:07:41.469787 9349 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:41.507995 9349 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:41.511257 9349 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:41.511453 9349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:41.512041 9349 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:41.512969 9349 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:41.513018 9349 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:41.513070 9349 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:41.513095 9349 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:41.522076 9349 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:38001
I20260203 21:07:41.522099 9477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:38001 every 8 connection(s)
I20260203 21:07:41.523057 9349 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:07:41.525727 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 9349
I20260203 21:07:41.525871 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:07:41.526146 410 external_mini_cluster.cc:1468] Setting key 15b02abe9841d4488950a48cbde3b7d3
Feb 03 21:07:41 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.102.130@KRBTEST.COM; defaulting to no policy
I20260203 21:07:41.536798 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:41.524946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:48641 (local address 127.0.102.190:46645)
0203 21:07:41.525079 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:41.525083 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:41.525840 (+ 757us) server_negotiation.cc:408] Connection header received
0203 21:07:41.526839 (+ 999us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:41.526843 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:41.526925 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:41.526997 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:41.529165 (+ 2168us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:41.529938 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:41.530615 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:41.530801 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:41.533078 (+ 2277us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:41.533106 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:41.533114 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:41.533147 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:41.534681 (+ 1534us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:41.535230 (+ 549us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:41.535234 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:41.535235 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:41.535286 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:41.535562 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:41.535566 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:41.535568 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:41.535751 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:41.535843 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:41.536454 (+ 611us) server_negotiation.cc:300] Negotiation successful
0203 21:07:41.536595 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260203 21:07:41.537570 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:41.525159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:48641)
0203 21:07:41.525666 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:41.525718 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:41.526655 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:41.527195 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:41.527207 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:41.527682 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:41.528990 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:41.529011 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:41.530088 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:41.530091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:41.530504 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:41.530511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:41.530715 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:41.531274 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:41.531303 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:41.532911 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:41.534831 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:41.534836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:41.534848 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:41.535106 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:41.535373 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:41.535376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:41.535378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:41.535474 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:41.535857 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:41.535865 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:41.536186 (+ 321us) client_negotiation.cc:769] Sending connection context
0203 21:07:41.536410 (+ 224us) client_negotiation.cc:240] Negotiation successful
0203 21:07:41.536740 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.130@KRBTEST.COM" created.
I20260203 21:07:41.538800 9478 heartbeater.cc:344] Connected to a master server at 127.0.102.190:46645
I20260203 21:07:41.539099 9478 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:41.539673 9478 heartbeater.cc:507] Master 127.0.102.190:46645 requested a full tablet report, sending...
I20260203 21:07:41.541517 9285 ts_manager.cc:194] Registered new tserver with Master: e7d63ca3af004e71a32911a73cfd3457 (127.0.102.129:38001)
I20260203 21:07:41.543174 9285 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:48641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:07:41.579892 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:46645
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:41.687471 9485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:41.687713 9485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:41.687772 9485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:41.691170 9485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:41.691247 9485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:41.691346 9485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:07:41.695794 9485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:46645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:41.696992 9485 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:41.697855 9485 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:41.705058 9493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:41.705058 9490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:41.705284 9491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:41.705353 9485 server_base.cc:1060] running on GCE node
I20260203 21:07:41.705722 9485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:41.706290 9485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:41.707474 9485 hybrid_clock.cc:648] HybridClock initialized: now 1770152861707456 us; error 32 us; skew 500 ppm
Feb 03 21:07:41 dist-test-slave-j889 krb5kdc[9237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:41.710637 9485 init.cc:377] Logged in from keytab as kudu/127.0.102.130@KRBTEST.COM (short username kudu)
I20260203 21:07:41.711851 9485 webserver.cc:492] Webserver started at http://127.0.102.130:36299/ using document root <none> and password file <none>
I20260203 21:07:41.712431 9485 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:41.712486 9485 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:41.712729 9485 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:41.714490 9485 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "399abbd8550b453a99b96325ab3e43dc"
format_stamp: "Formatted at 2026-02-03 21:07:41 on dist-test-slave-j889"
server_key: "de608b66d234ece42d1315b4780786e9"
server_key_iv: "414c4d011c46b35134a30b3077ec4c5b"
server_key_version: "encryptionkey@0"
I20260203 21:07:41.714964 9485 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "399abbd8550b453a99b96325ab3e43dc"
format_stamp: "Formatted at 2026-02-03 21:07:41 on dist-test-slave-j889"
server_key: "de608b66d234ece42d1315b4780786e9"
server_key_iv: "414c4d011c46b35134a30b3077ec4c5b"
server_key_version: "encryptionkey@0"
I20260203 21:07:41.718590 9485 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260203 21:07:41.721055 9500 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:41.722208 9485 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:07:41.722311 9485 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "399abbd8550b453a99b96325ab3e43dc"
format_stamp: "Formatted at 2026-02-03 21:07:41 on dist-test-slave-j889"
server_key: "de608b66d234ece42d1315b4780786e9"
server_key_iv: "414c4d011c46b35134a30b3077ec4c5b"
server_key_version: "encryptionkey@0"
I20260203 21:07:41.722396 9485 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:41.731935 9485 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:41.734920 9485 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:41.735077 9485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:41.735635 9485 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:41.736496 9485 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:41.736544 9485 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:41.736583 9485 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:41.736603 9485 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:41.745991 9485 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:41407
I20260203 21:07:41.746013 9613 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:41407 every 8 connection(s)
I20260203 21:07:41.747083 9485 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
Feb 03 21:07:41 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:07:41.755887 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 9485
I20260203 21:07:41.756021 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:07:41.756283 410 external_mini_cluster.cc:1468] Setting key f44aa14cf81ec6ce07393f9e522dacc3
I20260203 21:07:41.759894 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:41.748911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:55483 (local address 127.0.102.190:46645)
0203 21:07:41.749035 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:41.749038 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:41.749868 (+ 830us) server_negotiation.cc:408] Connection header received
0203 21:07:41.750777 (+ 909us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:41.750781 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:41.750844 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:41.750919 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:41.752717 (+ 1798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:41.753215 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:41.753858 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:41.754040 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:41.756352 (+ 2312us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:41.756375 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:41.756378 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:41.756409 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:41.757913 (+ 1504us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:41.758452 (+ 539us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:41.758455 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:41.758456 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:41.758496 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:41.758764 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:41.758766 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:41.758768 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:41.758908 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:41.758981 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:41.759592 (+ 611us) server_negotiation.cc:300] Negotiation successful
0203 21:07:41.759732 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260203 21:07:41.760535 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:41.749215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:55483)
0203 21:07:41.749703 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:41.749744 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:41.750571 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:41.751098 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:41.751108 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:41.751539 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:41.752549 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:41.752565 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:41.753335 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:41.753338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:41.753747 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:41.753754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:41.753983 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:41.754541 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:41.754569 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:41.756175 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:41.758033 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:41.758039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:41.758052 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:41.758363 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:41.758564 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:41.758567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:41.758569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:41.758687 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:41.758995 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:41.759001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:41.759326 (+ 325us) client_negotiation.cc:769] Sending connection context
0203 21:07:41.759533 (+ 207us) client_negotiation.cc:240] Negotiation successful
0203 21:07:41.759770 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":113,"threads_started":1}
I20260203 21:07:41.761653 9614 heartbeater.cc:344] Connected to a master server at 127.0.102.190:46645
I20260203 21:07:41.761901 9614 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:41.762420 9614 heartbeater.cc:507] Master 127.0.102.190:46645 requested a full tablet report, sending...
I20260203 21:07:41.763479 9285 ts_manager.cc:194] Registered new tserver with Master: 399abbd8550b453a99b96325ab3e43dc (127.0.102.130:41407)
I20260203 21:07:41.764048 9285 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.130@KRBTEST.COM'} at 127.0.102.130:55483
WARNING: no policy specified for kudu/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:07:41.807170 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:46645
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:41.916126 9621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:41.916366 9621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:41.916424 9621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:41.919759 9621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:41.919840 9621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:41.919922 9621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:07:41.924314 9621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:46645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:41.925447 9621 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:41.926306 9621 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:41.932958 9627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:41.932950 9626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:41.932942 9629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:41.933259 9621 server_base.cc:1060] running on GCE node
I20260203 21:07:41.933645 9621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:41.934363 9621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:41.935549 9621 hybrid_clock.cc:648] HybridClock initialized: now 1770152861935517 us; error 48 us; skew 500 ppm
Feb 03 21:07:41 dist-test-slave-j889 krb5kdc[9237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:41.938309 9621 init.cc:377] Logged in from keytab as kudu/127.0.102.131@KRBTEST.COM (short username kudu)
I20260203 21:07:41.939380 9621 webserver.cc:492] Webserver started at http://127.0.102.131:36339/ using document root <none> and password file <none>
I20260203 21:07:41.939947 9621 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:41.940011 9621 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:41.940184 9621 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:41.941926 9621 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "f54930b60c7141159b241fa457406e5f"
format_stamp: "Formatted at 2026-02-03 21:07:41 on dist-test-slave-j889"
server_key: "0a779a8bb7b0594658239da7cdffc53e"
server_key_iv: "e2448b4f5738b10c70d14ca22e2a2b22"
server_key_version: "encryptionkey@0"
I20260203 21:07:41.942377 9621 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "f54930b60c7141159b241fa457406e5f"
format_stamp: "Formatted at 2026-02-03 21:07:41 on dist-test-slave-j889"
server_key: "0a779a8bb7b0594658239da7cdffc53e"
server_key_iv: "e2448b4f5738b10c70d14ca22e2a2b22"
server_key_version: "encryptionkey@0"
I20260203 21:07:41.945811 9621 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260203 21:07:41.948288 9636 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:41.949476 9621 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260203 21:07:41.949599 9621 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "f54930b60c7141159b241fa457406e5f"
format_stamp: "Formatted at 2026-02-03 21:07:41 on dist-test-slave-j889"
server_key: "0a779a8bb7b0594658239da7cdffc53e"
server_key_iv: "e2448b4f5738b10c70d14ca22e2a2b22"
server_key_version: "encryptionkey@0"
I20260203 21:07:41.949717 9621 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:41.978763 9621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:41.981688 9621 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:41.981861 9621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:41.982393 9621 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:41.983248 9621 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:41.983295 9621 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:41.983347 9621 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:41.983371 9621 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:41.992400 9621 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:36907
I20260203 21:07:41.992431 9749 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:36907 every 8 connection(s)
I20260203 21:07:41.993376 9621 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
Feb 03 21:07:41 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:07:42.003366 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 9621
I20260203 21:07:42.003487 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:07:42.003741 410 external_mini_cluster.cc:1468] Setting key 205db0a19d9a736c7209b78de7d5ef14
I20260203 21:07:42.005595 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:41.995124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:57435 (local address 127.0.102.190:46645)
0203 21:07:41.995293 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:41.995297 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:41.995986 (+ 689us) server_negotiation.cc:408] Connection header received
0203 21:07:41.996833 (+ 847us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:41.996836 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:41.996888 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:41.996974 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:41.998634 (+ 1660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:41.999150 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:41.999773 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:41.999918 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.002073 (+ 2155us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:42.002095 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:42.002097 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:42.002125 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:42.003393 (+ 1268us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:42.003960 (+ 567us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:42.003964 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:42.003966 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:42.004044 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:42.004387 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:42.004391 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:42.004392 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:42.004601 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:42.004731 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:42.005324 (+ 593us) server_negotiation.cc:300] Negotiation successful
0203 21:07:42.005445 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260203 21:07:42.006192 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:41.995366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:57435)
0203 21:07:41.995839 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:41.995875 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:41.996602 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:41.997142 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:41.997150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:41.997532 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:41.998483 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:41.998498 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:41.999269 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:41.999272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:41.999682 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:41.999689 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:41.999860 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.000363 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:42.000387 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:42.001916 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:42.003502 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:42.003508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:42.003520 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:42.003865 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:42.004139 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:07:42.004143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:07:42.004145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:07:42.004266 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:07:42.004735 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:42.004742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:42.005060 (+ 318us) client_negotiation.cc:769] Sending connection context
0203 21:07:42.005263 (+ 203us) client_negotiation.cc:240] Negotiation successful
0203 21:07:42.005485 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":108,"threads_started":1}
I20260203 21:07:42.007206 9750 heartbeater.cc:344] Connected to a master server at 127.0.102.190:46645
I20260203 21:07:42.007438 9750 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:42.007947 9750 heartbeater.cc:507] Master 127.0.102.190:46645 requested a full tablet report, sending...
I20260203 21:07:42.008909 9285 ts_manager.cc:194] Registered new tserver with Master: f54930b60c7141159b241fa457406e5f (127.0.102.131:36907)
I20260203 21:07:42.009456 9285 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.131@KRBTEST.COM'} at 127.0.102.131:57435
I20260203 21:07:42.017866 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/oryx_127.0.102.129.keytab.
Entry for principal oryx/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/oryx_127.0.102.129.keytab.
WARNING: no policy specified for oryx/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/oryx_127.0.102.130.keytab.
Entry for principal oryx/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/oryx_127.0.102.130.keytab.
WARNING: no policy specified for oryx/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/oryx_127.0.102.131.keytab.
Entry for principal oryx/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/oryx_127.0.102.131.keytab.
I20260203 21:07:42.087499 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 9349
I20260203 21:07:42.092438 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 9485
I20260203 21:07:42.097223 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 9621
I20260203 21:07:42.101965 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 9253
I20260203 21:07:42.107086 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:46645
--webserver_interface=127.0.102.190
--webserver_port=42763
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:46645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:07:42.213821 9759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:42.214058 9759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:42.214135 9759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:42.217363 9759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:07:42.217427 9759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:42.217458 9759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:42.217481 9759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:07:42.217510 9759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:07:42.221715 9759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:46645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:46645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=42763
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:42.222752 9759 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:42.223560 9759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:42.229075 9764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:42.229167 9759 server_base.cc:1060] running on GCE node
W20260203 21:07:42.229264 9767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:42.229375 9765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:42.229920 9759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:42.230857 9759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:42.231999 9759 hybrid_clock.cc:648] HybridClock initialized: now 1770152862231982 us; error 37 us; skew 500 ppm
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:42.234563 9759 init.cc:377] Logged in from keytab as kudu/127.0.102.190@KRBTEST.COM (short username kudu)
I20260203 21:07:42.235652 9759 webserver.cc:492] Webserver started at http://127.0.102.190:42763/ using document root <none> and password file <none>
I20260203 21:07:42.236191 9759 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:42.236256 9759 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:42.239703 9759 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:42.241657 9774 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:42.242647 9759 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260203 21:07:42.242770 9759 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "b4a2dd4f57a74e97a84571b7a73818ba"
format_stamp: "Formatted at 2026-02-03 21:07:41 on dist-test-slave-j889"
server_key: "e445d7375fc54422e5dd65b09fbb3813"
server_key_iv: "279b3926a44a3a2b455d593065f2d4f0"
server_key_version: "encryptionkey@0"
I20260203 21:07:42.243171 9759 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:42.253720 9759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:42.256462 9759 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:42.256659 9759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:42.264112 9759 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:46645
I20260203 21:07:42.264109 9826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:46645 every 8 connection(s)
I20260203 21:07:42.265204 9759 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:07:42.269752 9827 sys_catalog.cc:263] Verifying existing consensus state
I20260203 21:07:42.271375 9827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba: Bootstrap starting.
I20260203 21:07:42.272714 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 9759
I20260203 21:07:42.273154 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:38001
--local_ip_for_outbound_sockets=127.0.102.129
--tserver_master_addrs=127.0.102.190:46645
--webserver_port=39615
--webserver_interface=127.0.102.129
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.102.129
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/oryx_127.0.102.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260203 21:07:42.287094 9827 log.cc:826] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:42.293275 9827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260203 21:07:42.293587 9827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba: Bootstrap complete.
I20260203 21:07:42.296860 9827 raft_consensus.cc:359] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a2dd4f57a74e97a84571b7a73818ba" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46645 } }
I20260203 21:07:42.297181 9827 raft_consensus.cc:740] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a2dd4f57a74e97a84571b7a73818ba, State: Initialized, Role: FOLLOWER
I20260203 21:07:42.297624 9827 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a2dd4f57a74e97a84571b7a73818ba" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46645 } }
I20260203 21:07:42.297753 9827 raft_consensus.cc:399] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:07:42.297814 9827 raft_consensus.cc:493] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:07:42.297901 9827 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 1 FOLLOWER]: Advancing to term 2
I20260203 21:07:42.299348 9827 raft_consensus.cc:515] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a2dd4f57a74e97a84571b7a73818ba" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46645 } }
I20260203 21:07:42.299664 9827 leader_election.cc:304] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4a2dd4f57a74e97a84571b7a73818ba; no voters:
I20260203 21:07:42.299928 9827 leader_election.cc:290] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [CANDIDATE]: Term 2 election: Requested vote from peers
I20260203 21:07:42.300037 9831 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 2 FOLLOWER]: Leader election won for term 2
I20260203 21:07:42.300220 9831 raft_consensus.cc:697] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [term 2 LEADER]: Becoming Leader. State: Replica: b4a2dd4f57a74e97a84571b7a73818ba, State: Running, Role: LEADER
I20260203 21:07:42.300501 9831 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a2dd4f57a74e97a84571b7a73818ba" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46645 } }
I20260203 21:07:42.301095 9827 sys_catalog.cc:565] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:07:42.301755 9832 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b4a2dd4f57a74e97a84571b7a73818ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a2dd4f57a74e97a84571b7a73818ba" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46645 } } }
I20260203 21:07:42.301869 9832 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:42.302069 9832 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4a2dd4f57a74e97a84571b7a73818ba. Latest consensus state: current_term: 2 leader_uuid: "b4a2dd4f57a74e97a84571b7a73818ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4a2dd4f57a74e97a84571b7a73818ba" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 46645 } } }
I20260203 21:07:42.302146 9832 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:42.302650 9837 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:07:42.305617 9837 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:07:42.306260 9837 catalog_manager.cc:1269] Loaded cluster ID: 7877b683d5904de7a324c1f35a566300
I20260203 21:07:42.306318 9837 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:07:42.308785 9837 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:07:42.309581 9837 catalog_manager.cc:6038] T 00000000000000000000000000000000 P b4a2dd4f57a74e97a84571b7a73818ba: Loaded TSK: 0
I20260203 21:07:42.310962 9837 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260203 21:07:42.381892 9829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:42.382141 9829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:42.382216 9829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:42.385581 9829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:42.385654 9829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:42.385762 9829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:07:42.390187 9829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/oryx_127.0.102.129.keytab
--principal=oryx/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:38001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=39615
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:46645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:42.391384 9829 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:42.392278 9829 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:42.398846 9854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:42.398846 9853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:42.398846 9856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:42.399266 9829 server_base.cc:1060] running on GCE node
I20260203 21:07:42.399607 9829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:42.400149 9829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:42.401304 9829 hybrid_clock.cc:648] HybridClock initialized: now 1770152862401293 us; error 27 us; skew 500 ppm
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152862, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:42.403749 9829 init.cc:377] Logged in from keytab as oryx/127.0.102.129@KRBTEST.COM (short username oryx)
I20260203 21:07:42.404803 9829 webserver.cc:492] Webserver started at http://127.0.102.129:39615/ using document root <none> and password file <none>
I20260203 21:07:42.405334 9829 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:42.405391 9829 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:42.408869 9829 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260203 21:07:42.410717 9863 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:42.411778 9829 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:42.411901 9829 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "e7d63ca3af004e71a32911a73cfd3457"
format_stamp: "Formatted at 2026-02-03 21:07:41 on dist-test-slave-j889"
server_key: "3f9a0094b26bfe62a37a8ea697c99df9"
server_key_iv: "76eb665684fd1f8837521aba47e5d3bc"
server_key_version: "encryptionkey@0"
I20260203 21:07:42.412308 9829 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:42.429235 9829 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:42.432819 9829 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:42.433032 9829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:42.433729 9829 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:42.434870 9829 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:42.434926 9829 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:42.434981 9829 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:42.435003 9829 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:42.445724 9829 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:38001
I20260203 21:07:42.445750 9976 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:38001 every 8 connection(s)
I20260203 21:07:42.446761 9829 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:07:42.448554 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 9829
I20260203 21:07:42.449071 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:41407
--local_ip_for_outbound_sockets=127.0.102.130
--tserver_master_addrs=127.0.102.190:46645
--webserver_port=36299
--webserver_interface=127.0.102.130
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.102.130
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/oryx_127.0.102.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:42.461323 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:42.449072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:34197)
0203 21:07:42.450398 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:42.450433 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:42.451229 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:42.452854 (+ 1625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:42.452865 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:42.453338 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:42.454514 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.454532 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.456119 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.456123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:42.456810 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.456821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.457045 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.457787 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:42.457830 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:42.460555 (+ 2725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1150,"thread_start_us":116,"threads_started":1}
W20260203 21:07:42.461454 9979 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:42.449013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:34197 (local address 127.0.102.190:46645)
0203 21:07:42.451683 (+ 2670us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:42.451695 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:42.451732 (+ 37us) server_negotiation.cc:408] Connection header received
0203 21:07:42.452013 (+ 281us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:42.452041 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:42.452400 (+ 359us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:42.452793 (+ 393us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:42.454707 (+ 1914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.455953 (+ 1246us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.456931 (+ 978us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.457225 (+ 294us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.460809 (+ 3584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:34197: BlockingRecv error: recv got EOF from 127.0.102.129:34197 (error 108)
Metrics: {"server-negotiator.queue_time_us":2522,"thread_start_us":121,"threads_started":1}
W20260203 21:07:42.461982 9977 heartbeater.cc:646] Failed to heartbeat to 127.0.102.190:46645 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.102.190:46645: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:42.467460 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:42.462415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:43539)
0203 21:07:42.462518 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:42.462532 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:42.462609 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:42.462908 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:42.462911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:42.463064 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:42.463290 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.463296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.464218 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.464222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:42.464821 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.464829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.464927 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.465394 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:42.465408 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:42.467333 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260203 21:07:42.467587 9979 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:42.462590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:43539 (local address 127.0.102.190:46645)
0203 21:07:42.462702 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:42.462707 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:42.462719 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:42.462756 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:42.462759 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:42.462807 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:42.462890 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:42.463400 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.464115 (+ 715us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.464918 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.465415 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.467476 (+ 2061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:43539: BlockingRecv error: recv got EOF from 127.0.102.129:43539 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:42.473271 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:42.468221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:42751)
0203 21:07:42.468382 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:42.468395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:42.468471 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:42.468718 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:42.468722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:42.468914 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:42.469124 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.469131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.470085 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.470089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:42.470572 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.470580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.470685 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.471203 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:42.471218 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:42.473173 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95}
W20260203 21:07:42.473453 9977 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260203 21:07:42.473600 9979 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:42.468292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:42751 (local address 127.0.102.190:46645)
0203 21:07:42.468394 (+ 102us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:42.468398 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:42.468433 (+ 35us) server_negotiation.cc:408] Connection header received
0203 21:07:42.468549 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:42.468553 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:42.468597 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:42.468707 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:42.469237 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.469959 (+ 722us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.470679 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.471177 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.473319 (+ 2142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:42751: BlockingRecv error: recv got EOF from 127.0.102.129:42751 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260203 21:07:42.559350 9981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:42.559577 9981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:42.559634 9981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:42.562840 9981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:42.562906 9981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:42.562983 9981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:07:42.567196 9981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/oryx_127.0.102.130.keytab
--principal=oryx/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:41407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=36299
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:46645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:42.568229 9981 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:42.569082 9981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:42.575775 9986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:42.575795 9989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:42.575785 9987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:42.576107 9981 server_base.cc:1060] running on GCE node
I20260203 21:07:42.576439 9981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:42.577018 9981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:42.578198 9981 hybrid_clock.cc:648] HybridClock initialized: now 1770152862578169 us; error 36 us; skew 500 ppm
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152862, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:42.580734 9981 init.cc:377] Logged in from keytab as oryx/127.0.102.130@KRBTEST.COM (short username oryx)
I20260203 21:07:42.581856 9981 webserver.cc:492] Webserver started at http://127.0.102.130:36299/ using document root <none> and password file <none>
I20260203 21:07:42.582410 9981 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:42.582474 9981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:42.586071 9981 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:07:42.587906 9996 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:42.588912 9981 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:07:42.589012 9981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "399abbd8550b453a99b96325ab3e43dc"
format_stamp: "Formatted at 2026-02-03 21:07:41 on dist-test-slave-j889"
server_key: "de608b66d234ece42d1315b4780786e9"
server_key_iv: "414c4d011c46b35134a30b3077ec4c5b"
server_key_version: "encryptionkey@0"
I20260203 21:07:42.589424 9981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:42.601553 9981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:42.604060 9981 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:42.604215 9981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:42.604741 9981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:42.605576 9981 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:42.605621 9981 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:42.605672 9981 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:42.605700 9981 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:42.614452 9981 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:41407
I20260203 21:07:42.614471 10109 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:41407 every 8 connection(s)
I20260203 21:07:42.615468 9981 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
W20260203 21:07:42.624401 9979 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:42.617189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:52279 (local address 127.0.102.190:46645)
0203 21:07:42.617333 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:42.617338 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:42.618014 (+ 676us) server_negotiation.cc:408] Connection header received
0203 21:07:42.618798 (+ 784us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:42.618802 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:42.618852 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:42.618924 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:42.620476 (+ 1552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.621007 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.621603 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.621723 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.624303 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:52279: BlockingRecv error: recv got EOF from 127.0.102.130:52279 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260203 21:07:42.624845 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:42.617459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:52279)
0203 21:07:42.617877 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:42.617913 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:42.618627 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:42.619075 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:42.619083 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:42.619450 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:42.620362 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.620375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.621114 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.621117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:42.621521 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.621527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.621690 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.622220 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:42.622244 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:42.624186 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":100,"threads_started":1}
I20260203 21:07:42.624898 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 9981
W20260203 21:07:42.625241 10110 heartbeater.cc:646] Failed to heartbeat to 127.0.102.190:46645 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.102.190:46645: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
I20260203 21:07:42.625326 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:36907
--local_ip_for_outbound_sockets=127.0.102.131
--tserver_master_addrs=127.0.102.190:46645
--webserver_port=36339
--webserver_interface=127.0.102.131
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:35577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.102.131
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/oryx_127.0.102.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:42.629657 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:42.625618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:56503)
0203 21:07:42.625760 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:42.625774 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:42.625859 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:42.626061 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:42.626064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:42.626202 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:42.626359 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.626364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.627009 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.627012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:42.627369 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.627376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.627471 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.627965 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:42.627981 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:42.629544 (+ 1563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260203 21:07:42.629760 9979 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:42.625718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:56503 (local address 127.0.102.190:46645)
0203 21:07:42.625834 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:42.625838 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:42.625850 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:42.625933 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:42.625937 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:42.625977 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:42.626037 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:42.626463 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.626914 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.627466 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.627633 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.629666 (+ 2033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:56503: BlockingRecv error: recv got EOF from 127.0.102.130:56503 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:42.634131 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:42.630180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:59295)
0203 21:07:42.630276 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:42.630290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:42.630358 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:42.630559 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:42.630562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:42.630688 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:42.630836 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.630841 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.631483 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.631486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:42.631831 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.631837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.631931 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.632394 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:42.632406 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:42.634013 (+ 1607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260203 21:07:42.634301 10110 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260203 21:07:42.634223 9979 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:42.630270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:59295 (local address 127.0.102.190:46645)
0203 21:07:42.630386 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:42.630390 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:42.630403 (+ 13us) server_negotiation.cc:408] Connection header received
0203 21:07:42.630435 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:42.630438 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:42.630476 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:42.630537 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:42.630938 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.631390 (+ 452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.631925 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.632110 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.634122 (+ 2012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:59295: BlockingRecv error: recv got EOF from 127.0.102.130:59295 (error 108)
Metrics: {"server-negotiator.queue_time_us":59,"thread_start_us":81,"threads_started":1}
W20260203 21:07:42.727514 10113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:42.727716 10113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:42.727780 10113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:42.730950 10113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:42.731014 10113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:42.731117 10113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:07:42.735123 10113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:35577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/krb5kdc/oryx_127.0.102.131.keytab
--principal=oryx/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:36907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=36339
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:46645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:42.736099 10113 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:42.736920 10113 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:42.743091 10119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:42.743067 10122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:42.743086 10120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:42.743244 10113 server_base.cc:1060] running on GCE node
I20260203 21:07:42.743593 10113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:42.744089 10113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:42.745225 10113 hybrid_clock.cc:648] HybridClock initialized: now 1770152862745204 us; error 34 us; skew 500 ppm
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152862, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:07:42.747514 10113 init.cc:377] Logged in from keytab as oryx/127.0.102.131@KRBTEST.COM (short username oryx)
I20260203 21:07:42.748493 10113 webserver.cc:492] Webserver started at http://127.0.102.131:36339/ using document root <none> and password file <none>
I20260203 21:07:42.749053 10113 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:42.749117 10113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:42.752457 10113 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:07:42.754309 10129 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:42.755290 10113 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260203 21:07:42.755391 10113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "f54930b60c7141159b241fa457406e5f"
format_stamp: "Formatted at 2026-02-03 21:07:41 on dist-test-slave-j889"
server_key: "0a779a8bb7b0594658239da7cdffc53e"
server_key_iv: "e2448b4f5738b10c70d14ca22e2a2b22"
server_key_version: "encryptionkey@0"
I20260203 21:07:42.755792 10113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:42.767344 10113 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:42.769716 10113 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:42.769886 10113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:42.770378 10113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:42.771188 10113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:42.771234 10113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:42.771281 10113 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:42.771309 10113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:42.779718 10113 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:36907
I20260203 21:07:42.779739 10242 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:36907 every 8 connection(s)
I20260203 21:07:42.780742 10113 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
W20260203 21:07:42.789680 9979 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:42.782317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:46149 (local address 127.0.102.190:46645)
0203 21:07:42.782457 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:42.782461 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:42.783090 (+ 629us) server_negotiation.cc:408] Connection header received
0203 21:07:42.783895 (+ 805us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:42.783899 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:42.783956 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:42.784026 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:42.785652 (+ 1626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.786186 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.786810 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.786986 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.789585 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:46149: BlockingRecv error: recv got EOF from 127.0.102.131:46149 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260203 21:07:42.790041 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 10113
I20260203 21:07:42.790143 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:42.782571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:46149)
0203 21:07:42.782958 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:42.782993 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:42.783696 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:42.784198 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:42.784206 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:42.784571 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:42.785499 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.785514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.786299 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.786302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:42.786699 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.786706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.786912 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.787451 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:42.787476 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:42.789464 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":99,"threads_started":1}
W20260203 21:07:42.790517 10243 heartbeater.cc:646] Failed to heartbeat to 127.0.102.190:46645 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.102.190:46645: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
W20260203 21:07:42.795660 10114 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:42.791083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:50823 (local address 127.0.102.190:46645)
0203 21:07:42.791191 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:42.791194 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:42.791204 (+ 10us) server_negotiation.cc:408] Connection header received
0203 21:07:42.791236 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:42.791239 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:42.791301 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:42.791386 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:42.791806 (+ 420us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.792449 (+ 643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.793119 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.793294 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.795556 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:50823: BlockingRecv error: recv got EOF from 127.0.102.131:50823 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260203 21:07:42.795663 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:42.790891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:50823)
0203 21:07:42.790998 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:42.791012 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:42.791091 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:42.791416 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:42.791418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:42.791539 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:42.791699 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.791705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.792555 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.792559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:42.793021 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.793029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.793114 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.793575 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:42.793590 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:42.795424 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
I20260203 21:07:42.798352 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:42.790905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.0.102.190:46645)
0203 21:07:42.791012 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:42.791016 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:42.791028 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:07:42.791134 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:42.791138 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:42.791182 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:42.791243 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:42.792155 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.792687 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.793580 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.793757 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.794670 (+ 913us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:42.794692 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:42.794709 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:07:42.794763 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:42.796539 (+ 1776us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:42.796985 (+ 446us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:42.796989 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:42.796993 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:42.797066 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:07:42.797285 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:07:42.797287 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:07:42.797288 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:07:42.797531 (+ 243us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:42.797699 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:42.797996 (+ 297us) server_negotiation.cc:300] Negotiation successful
0203 21:07:42.798223 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":69376}
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:42 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:42.800562 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:42.796206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:60159)
0203 21:07:42.796315 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:42.796328 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:42.796383 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:42.797015 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:42.797018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:42.797157 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:42.797346 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.797352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.797968 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.797971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:42.798340 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:42.798349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.798448 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.798849 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:42.798861 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:42.800464 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260203 21:07:42.800694 10114 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:42.796704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:60159 (local address 127.0.102.190:46645)
0203 21:07:42.796841 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:42.796845 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:42.796858 (+ 13us) server_negotiation.cc:408] Connection header received
0203 21:07:42.796887 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:42.796890 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:42.796933 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:42.796999 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:42.797439 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.797879 (+ 440us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:42.798444 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:42.798615 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:42.800567 (+ 1952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:60159: BlockingRecv error: recv got EOF from 127.0.102.131:60159 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260203 21:07:42.801002 10243 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Feb 03 21:07:43 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:43 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:43.479048 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:43.474182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:48849)
0203 21:07:43.474456 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:43.474471 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:43.474566 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:43.474932 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:43.474935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:43.475161 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:43.475393 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:43.475404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:43.476203 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:43.476206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:43.476601 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:43.476608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:43.476734 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:43.477271 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:43.477286 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:43.478927 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
W20260203 21:07:43.479175 10248 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:43.474282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:48849 (local address 127.0.102.190:46645)
0203 21:07:43.474538 (+ 256us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:43.474542 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:43.474555 (+ 13us) server_negotiation.cc:408] Connection header received
0203 21:07:43.474655 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:43.474658 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:43.474706 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:43.474801 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:43.475510 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:43.476091 (+ 581us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:43.476734 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:43.476917 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:43.479062 (+ 2145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:48849: BlockingRecv error: recv got EOF from 127.0.102.129:48849 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
Feb 03 21:07:43 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:43 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:43.640406 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:43.635039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:47605)
0203 21:07:43.635346 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:43.635360 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:43.635478 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:43.635819 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:43.635823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:43.636053 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:43.636257 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:43.636265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:43.636986 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:43.636989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:43.637387 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:43.637394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:43.637518 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:43.638161 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:43.638178 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:43.640277 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":76,"threads_started":1}
W20260203 21:07:43.640527 10248 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:43.635205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:47605 (local address 127.0.102.190:46645)
0203 21:07:43.635356 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:43.635360 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:43.635454 (+ 94us) server_negotiation.cc:408] Connection header received
0203 21:07:43.635599 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:43.635603 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:43.635663 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:43.635741 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:43.636363 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:43.636879 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:43.637530 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:43.637734 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:43.640422 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:47605: BlockingRecv error: recv got EOF from 127.0.102.130:47605 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Feb 03 21:07:43 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:43 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:43.806772 10250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:43.801624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:47619)
0203 21:07:43.801868 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:43.801903 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:43.802034 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:43.802323 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:43.802327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:43.802527 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:43.802729 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:43.802736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:43.803596 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:43.803599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:43.804032 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:43.804041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:43.804141 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:43.804683 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:43.804697 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:43.806633 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
W20260203 21:07:43.806936 10248 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:43.801730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:47619 (local address 127.0.102.190:46645)
0203 21:07:43.801871 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:43.801876 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:43.801962 (+ 86us) server_negotiation.cc:408] Connection header received
0203 21:07:43.802120 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:43.802123 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:43.802169 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:43.802228 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:43.802854 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:43.803473 (+ 619us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:43.804156 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:43.804306 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:43.806805 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:47619: BlockingRecv error: recv got EOF from 127.0.102.131:47619 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Feb 03 21:07:44 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:44 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:44.489125 10251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:44.479792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:46341)
0203 21:07:44.480068 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:44.480082 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:44.480177 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:44.480460 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:44.480463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:44.480669 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:44.480884 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:44.480890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:44.481639 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:44.481641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:44.482011 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:44.482017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:44.482098 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:44.487236 (+ 5138us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:44.487251 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:44.488993 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":80,"threads_started":1}
W20260203 21:07:44.489274 10252 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:44.479898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:46341 (local address 127.0.102.190:46645)
0203 21:07:44.480135 (+ 237us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:44.480138 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:44.480152 (+ 14us) server_negotiation.cc:408] Connection header received
0203 21:07:44.480268 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:44.480270 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:44.480313 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:44.480392 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:44.481038 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:44.481553 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:44.482092 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:44.482223 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:44.489146 (+ 6923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:46341: BlockingRecv error: recv got EOF from 127.0.102.129:46341 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":86,"threads_started":1}
Feb 03 21:07:44 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:44 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:44.645709 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:44.641106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:52505)
0203 21:07:44.641404 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:44.641417 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:44.641528 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:44.641767 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:44.641770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:44.641967 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:44.642158 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:44.642165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:44.642865 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:44.642870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:44.643242 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:44.643249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:44.643365 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:44.643865 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:44.643878 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:44.645562 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":86,"threads_started":1}
W20260203 21:07:44.645792 10252 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:44.641192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:52505 (local address 127.0.102.190:46645)
0203 21:07:44.641335 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:44.641340 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:44.641486 (+ 146us) server_negotiation.cc:408] Connection header received
0203 21:07:44.641616 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:44.641619 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:44.641666 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:44.641732 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:44.642278 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:44.642764 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:44.643364 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:44.643542 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:44.645694 (+ 2152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:52505: BlockingRecv error: recv got EOF from 127.0.102.130:52505 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Feb 03 21:07:44 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:44 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:44.812322 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:44.807647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:50933)
0203 21:07:44.807940 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:44.807953 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:44.808055 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:44.808297 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:44.808300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:44.808481 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:44.808725 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:44.808732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:44.809406 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:44.809409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:44.809796 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:44.809802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:44.809885 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:44.810359 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:44.810371 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:44.812198 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":97,"threads_started":1}
W20260203 21:07:44.812428 10252 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:44.807779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:50933 (local address 127.0.102.190:46645)
0203 21:07:44.807921 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:44.807924 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:44.808017 (+ 93us) server_negotiation.cc:408] Connection header received
0203 21:07:44.808136 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:44.808138 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:44.808185 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:44.808252 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:44.808824 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:44.809328 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:44.809873 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:44.809997 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:44.812313 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:50933: BlockingRecv error: recv got EOF from 127.0.102.131:50933 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Feb 03 21:07:45 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:45 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:45.494753 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:45.489804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:39393)
0203 21:07:45.490072 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:45.490086 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:45.490191 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:45.490429 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:45.490442 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:45.490621 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:45.490824 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:45.490831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:45.491681 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:45.491690 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:45.492085 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:45.492093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:45.492188 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:45.492738 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:45.492752 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:45.494608 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
W20260203 21:07:45.494863 10256 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:45.489917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:39393 (local address 127.0.102.190:46645)
0203 21:07:45.490130 (+ 213us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:45.490134 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:45.490147 (+ 13us) server_negotiation.cc:408] Connection header received
0203 21:07:45.490273 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:45.490276 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:45.490320 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:45.490397 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:45.490922 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:45.491572 (+ 650us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:45.492205 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:45.492357 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:45.494751 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:39393: BlockingRecv error: recv got EOF from 127.0.102.129:39393 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":54,"threads_started":1}
Feb 03 21:07:45 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:45 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:45.651420 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:45.646554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:47623)
0203 21:07:45.646831 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:45.646845 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:45.646959 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:45.647219 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:45.647221 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:45.647416 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:45.647630 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:45.647636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:45.648457 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:45.648461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:45.648900 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:45.648908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:45.649000 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:45.649477 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:45.649492 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:45.651271 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
W20260203 21:07:45.651521 10256 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:45.646660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:47623 (local address 127.0.102.190:46645)
0203 21:07:45.646787 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:45.646790 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:45.646907 (+ 117us) server_negotiation.cc:408] Connection header received
0203 21:07:45.647056 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:45.647058 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:45.647107 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:45.647199 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:45.647802 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:45.648320 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:45.648988 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:45.649116 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:45.651421 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:47623: BlockingRecv error: recv got EOF from 127.0.102.130:47623 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Feb 03 21:07:45 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:45 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:45.818064 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:45.813017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:38725)
0203 21:07:45.813291 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:45.813305 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:45.813415 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:45.813667 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:45.813670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:45.813847 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:45.814044 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:45.814050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:45.814798 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:45.814802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:45.815214 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:45.815223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:45.815316 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:45.815823 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:45.815836 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:45.817938 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
W20260203 21:07:45.818233 10256 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:45.813134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:38725 (local address 127.0.102.190:46645)
0203 21:07:45.813287 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:45.813291 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:45.813394 (+ 103us) server_negotiation.cc:408] Connection header received
0203 21:07:45.813493 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:45.813496 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:45.813546 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:45.813622 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:45.814172 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:45.814682 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:45.815333 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:45.815499 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:45.818107 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:38725: BlockingRecv error: recv got EOF from 127.0.102.131:38725 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Feb 03 21:07:46 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:46 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:46.500711 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:46.495642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:36367)
0203 21:07:46.495895 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:46.495908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:46.495997 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:46.496279 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:46.496282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:46.496483 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:46.496744 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:46.496753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:46.497671 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:46.497675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:46.498071 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:46.498078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:46.498182 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:46.498697 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:46.498711 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:46.500540 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
W20260203 21:07:46.500924 10260 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:46.495738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:36367 (local address 127.0.102.190:46645)
0203 21:07:46.495987 (+ 249us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:46.495991 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:46.496005 (+ 14us) server_negotiation.cc:408] Connection header received
0203 21:07:46.496078 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:46.496081 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:46.496124 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:46.496203 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:46.496903 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:46.497528 (+ 625us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:46.498203 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:46.498367 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:46.500775 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:36367: BlockingRecv error: recv got EOF from 127.0.102.129:36367 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
Feb 03 21:07:46 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:46 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:46.657464 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:46.652331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:60091)
0203 21:07:46.652691 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:46.652707 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:46.652824 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:46.653064 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:46.653066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:46.653258 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:46.653458 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:46.653465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:46.654198 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:46.654202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:46.654588 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:46.654594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:46.654697 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:46.655263 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:46.655278 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:46.657282 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":100,"threads_started":1}
W20260203 21:07:46.657564 10260 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:46.652450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:60091 (local address 127.0.102.190:46645)
0203 21:07:46.652667 (+ 217us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:46.652673 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:46.652780 (+ 107us) server_negotiation.cc:408] Connection header received
0203 21:07:46.652912 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:46.652916 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:46.652964 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:46.653028 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:46.653583 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:46.654094 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:46.654693 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:46.654875 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:46.657450 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:60091: BlockingRecv error: recv got EOF from 127.0.102.130:60091 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Feb 03 21:07:46 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:46 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:46.824210 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:46.818814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:60921)
0203 21:07:46.819063 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:46.819076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:46.819168 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:46.819435 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:46.819437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:46.819621 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:46.819815 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:46.819822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:46.820908 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:46.820911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:46.821332 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:46.821340 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:46.821453 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:46.821959 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:46.821972 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:46.824027 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":84,"threads_started":1}
W20260203 21:07:46.824488 10260 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:46.818908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:60921 (local address 127.0.102.190:46645)
0203 21:07:46.819047 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:46.819051 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:46.819126 (+ 75us) server_negotiation.cc:408] Connection header received
0203 21:07:46.819256 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:46.819259 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:46.819317 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:46.819413 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:46.819934 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:46.820792 (+ 858us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:46.821435 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:46.821652 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:46.824199 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:60921: BlockingRecv error: recv got EOF from 127.0.102.131:60921 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Feb 03 21:07:47 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:47 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:47.506664 10263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:47.501517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:33881)
0203 21:07:47.501805 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:47.501819 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:47.501913 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:47.502235 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:47.502238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:47.502459 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:47.502724 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:47.502734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:47.503583 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:47.503587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:47.504003 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:47.504012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:47.504114 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:47.504658 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:47.504676 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:47.506540 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
W20260203 21:07:47.506897 10264 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:47.501630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:33881 (local address 127.0.102.190:46645)
0203 21:07:47.501879 (+ 249us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:47.501882 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:47.501896 (+ 14us) server_negotiation.cc:408] Connection header received
0203 21:07:47.502030 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:47.502033 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:47.502090 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:47.502175 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:47.502862 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:47.503446 (+ 584us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:47.504133 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:47.504291 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:47.506761 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:33881: BlockingRecv error: recv got EOF from 127.0.102.129:33881 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
Feb 03 21:07:47 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:47 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:47.663021 10265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:47.658291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:58279)
0203 21:07:47.658585 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:47.658599 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:47.658708 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:47.658921 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:47.658923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:47.659102 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:47.659295 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:47.659300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:47.660054 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:47.660058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:47.660463 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:47.660471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:47.660570 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:47.661112 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:47.661125 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:47.662865 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
W20260203 21:07:47.663093 10264 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:47.658409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:58279 (local address 127.0.102.190:46645)
0203 21:07:47.658535 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:47.658539 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:47.658660 (+ 121us) server_negotiation.cc:408] Connection header received
0203 21:07:47.658779 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:47.658782 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:47.658827 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:47.658895 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:47.659384 (+ 489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:47.659924 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:47.660563 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:47.660734 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:47.662996 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:58279: BlockingRecv error: recv got EOF from 127.0.102.130:58279 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Feb 03 21:07:47 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:47 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:47.830463 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:47.825112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:41547)
0203 21:07:47.825459 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:47.825475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:47.825592 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:47.825856 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:47.825859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:47.826061 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:47.826296 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:47.826304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:47.827117 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:47.827121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:47.827551 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:47.827559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:47.827664 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:47.828261 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:47.828277 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:47.830278 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":86,"threads_started":1}
W20260203 21:07:47.830540 10264 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:47.825245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:41547 (local address 127.0.102.190:46645)
0203 21:07:47.825419 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:47.825424 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:47.825548 (+ 124us) server_negotiation.cc:408] Connection header received
0203 21:07:47.825697 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:47.825701 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:47.825748 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:47.825818 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:47.826472 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:47.826991 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:47.827662 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:47.827854 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:47.830437 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:41547: BlockingRecv error: recv got EOF from 127.0.102.131:41547 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Feb 03 21:07:48 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:48 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:48.512557 10267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:48.507486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:60179)
0203 21:07:48.507826 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:48.507841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:48.507940 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:48.508218 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:48.508220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:48.508425 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:48.508665 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:48.508674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:48.509512 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:48.509515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:48.509946 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:48.509954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:48.510048 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:48.510577 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:48.510591 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:48.512407 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":103,"threads_started":1}
W20260203 21:07:48.512713 10268 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:48.507614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:60179 (local address 127.0.102.190:46645)
0203 21:07:48.507885 (+ 271us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:48.507889 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:48.507905 (+ 16us) server_negotiation.cc:408] Connection header received
0203 21:07:48.508019 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:48.508021 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:48.508063 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:48.508143 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:48.508819 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:48.509412 (+ 593us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:48.510069 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:48.510222 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:48.512573 (+ 2351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:60179: BlockingRecv error: recv got EOF from 127.0.102.129:60179 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":84,"threads_started":1}
Feb 03 21:07:48 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:48 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:48.669088 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:48.663868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:37119)
0203 21:07:48.664171 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:48.664187 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:48.664322 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:48.664705 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:48.664709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:48.664916 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:48.665111 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:48.665117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:48.665853 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:48.665856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:48.666244 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:48.666250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:48.666351 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:48.666893 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:48.666908 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:48.668938 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":95,"threads_started":1}
W20260203 21:07:48.669224 10268 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:48.664000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:37119 (local address 127.0.102.190:46645)
0203 21:07:48.664182 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:48.664194 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:48.664300 (+ 106us) server_negotiation.cc:408] Connection header received
0203 21:07:48.664507 (+ 207us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:48.664511 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:48.664562 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:48.664655 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:48.665235 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:48.665736 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:48.666345 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:48.666526 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:48.669077 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:37119: BlockingRecv error: recv got EOF from 127.0.102.130:37119 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Feb 03 21:07:48 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:48 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:48.836601 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:48.831409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:42713)
0203 21:07:48.831691 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:48.831707 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:48.831813 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:48.832098 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:48.832101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:48.832272 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:48.832506 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:48.832516 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:48.833469 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:48.833473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:48.833909 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:48.833919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:48.834036 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:48.834530 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:48.834543 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:48.836454 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
W20260203 21:07:48.836764 10268 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:48.831543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:42713 (local address 127.0.102.190:46645)
0203 21:07:48.831726 (+ 183us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:48.831730 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:48.831765 (+ 35us) server_negotiation.cc:408] Connection header received
0203 21:07:48.831938 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:48.831942 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:48.831994 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:48.832078 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:48.832698 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:48.833360 (+ 662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:48.834034 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:48.834209 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:48.836594 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:42713: BlockingRecv error: recv got EOF from 127.0.102.131:42713 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Feb 03 21:07:49 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:49 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:49.521685 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:49.513397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:58733)
0203 21:07:49.513748 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:49.513763 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:49.513872 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:49.514139 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:49.514142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:49.514319 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:49.514515 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:49.514521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:49.515341 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:49.515344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:49.515724 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:49.515731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:49.515845 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:49.519475 (+ 3630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:49.519491 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:49.521537 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":97,"threads_started":1}
W20260203 21:07:49.521842 10272 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:49.513512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:58733 (local address 127.0.102.190:46645)
0203 21:07:49.513778 (+ 266us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:49.513782 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:49.513866 (+ 84us) server_negotiation.cc:408] Connection header received
0203 21:07:49.513979 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:49.513982 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:49.514032 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:49.514112 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:49.514641 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:49.515229 (+ 588us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:49.515854 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:49.516011 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:49.521696 (+ 5685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:58733: BlockingRecv error: recv got EOF from 127.0.102.129:58733 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
Feb 03 21:07:49 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:49 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:49.675076 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:49.669952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:35017)
0203 21:07:49.670328 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:49.670341 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:49.670458 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:49.670727 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:49.670730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:49.670963 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:49.671152 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:49.671158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:49.671966 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:49.671969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:49.672371 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:49.672378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:49.672480 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:49.673130 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:49.673146 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:49.674911 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":90,"threads_started":1}
W20260203 21:07:49.675163 10272 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:49.670075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:35017 (local address 127.0.102.190:46645)
0203 21:07:49.670263 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:49.670268 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:49.670413 (+ 145us) server_negotiation.cc:408] Connection header received
0203 21:07:49.670564 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:49.670567 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:49.670614 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:49.670712 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:49.671276 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:49.671811 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:49.672477 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:49.672706 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:49.675062 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:35017: BlockingRecv error: recv got EOF from 127.0.102.130:35017 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Feb 03 21:07:49 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:49 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:49.843016 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:49.837532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:57003)
0203 21:07:49.837883 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:49.837897 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:49.838011 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:49.838300 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:49.838304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:49.838494 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:49.838765 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:49.838775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:49.839577 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:49.839582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:49.840004 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:49.840012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:49.840110 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:49.840690 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:49.840706 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:49.842857 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":90,"threads_started":1}
W20260203 21:07:49.843163 10272 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:49.837668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:57003 (local address 127.0.102.190:46645)
0203 21:07:49.837855 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:49.837859 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:49.838008 (+ 149us) server_negotiation.cc:408] Connection header received
0203 21:07:49.838104 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:49.838107 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:49.838158 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:49.838222 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:49.838912 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:49.839455 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:49.840117 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:49.840257 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:49.843050 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:57003: BlockingRecv error: recv got EOF from 127.0.102.131:57003 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Feb 03 21:07:50 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:50 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:50.528388 10275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:50.522621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:36959)
0203 21:07:50.522923 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:50.522939 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:50.523057 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:50.523434 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:50.523437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:50.523657 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:50.523857 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:50.523875 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:50.524834 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:50.524837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:50.525333 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:50.525343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:50.525461 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:50.526018 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:50.526034 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:50.528235 (+ 2201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
W20260203 21:07:50.528528 10276 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:50.522829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:36959 (local address 127.0.102.190:46645)
0203 21:07:50.523118 (+ 289us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:50.523122 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:50.523138 (+ 16us) server_negotiation.cc:408] Connection header received
0203 21:07:50.523179 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:50.523182 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:50.523225 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:50.523315 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:50.524029 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:50.524714 (+ 685us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:50.525452 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:50.525643 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:50.528408 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:36959: BlockingRecv error: recv got EOF from 127.0.102.129:36959 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":100,"threads_started":1}
Feb 03 21:07:50 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:50 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:50.681226 10277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:50.676003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:37223)
0203 21:07:50.676344 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:50.676359 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:50.676482 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:50.676759 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:50.676761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:50.676988 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:50.677184 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:50.677191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:50.678085 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:50.678089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:50.678471 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:50.678478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:50.678584 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:50.679128 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:50.679142 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:50.681079 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":136,"threads_started":1}
W20260203 21:07:50.681336 10276 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:50.676153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:37223 (local address 127.0.102.190:46645)
0203 21:07:50.676334 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:50.676339 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:50.676437 (+ 98us) server_negotiation.cc:408] Connection header received
0203 21:07:50.676575 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:50.676578 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:50.676649 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:50.676717 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:50.677313 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:50.677977 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:50.678581 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:50.678762 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:50.681239 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:37223: BlockingRecv error: recv got EOF from 127.0.102.130:37223 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Feb 03 21:07:50 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:50 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:50.849259 10278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:50.843949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:49327)
0203 21:07:50.844263 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:50.844279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:50.844402 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:50.844676 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:50.844679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:50.844873 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:50.845073 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:50.845088 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:50.845790 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:50.845794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:50.846293 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:50.846305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:50.846433 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:50.847004 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:50.847017 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:50.849109 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":107,"threads_started":1}
W20260203 21:07:50.849349 10276 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:50.844063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:49327 (local address 127.0.102.190:46645)
0203 21:07:50.844240 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:50.844245 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:50.844350 (+ 105us) server_negotiation.cc:408] Connection header received
0203 21:07:50.844495 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:50.844498 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:50.844546 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:50.844612 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:50.845205 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:50.845685 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:50.846424 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:50.846605 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:50.849247 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:49327: BlockingRecv error: recv got EOF from 127.0.102.131:49327 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Feb 03 21:07:51 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:51 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:51.534914 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:51.529281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:39281)
0203 21:07:51.529574 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:51.529589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:51.529698 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:51.530177 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:51.530180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:51.530368 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:51.530569 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:51.530576 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:51.531399 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:51.531403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:51.531855 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:51.531865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:51.531991 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:51.532622 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:51.532702 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:51.534787 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":99,"threads_started":1}
W20260203 21:07:51.535094 10280 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:51.529472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:39281 (local address 127.0.102.190:46645)
0203 21:07:51.529679 (+ 207us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:51.529683 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:51.529698 (+ 15us) server_negotiation.cc:408] Connection header received
0203 21:07:51.529933 (+ 235us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:51.529938 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:51.530011 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:51.530107 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:51.530718 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:51.531253 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:51.532024 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:51.532219 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:51.534963 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:39281: BlockingRecv error: recv got EOF from 127.0.102.129:39281 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
Feb 03 21:07:51 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:51 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:51.686730 10281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:51.682022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:48759)
0203 21:07:51.682340 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:51.682353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:51.682469 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:51.682706 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:51.682708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:51.682907 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:51.683099 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:51.683105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:51.683846 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:51.683850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:51.684234 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:51.684241 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:51.684371 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:51.684924 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:51.684937 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:51.686585 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":89,"threads_started":1}
W20260203 21:07:51.686829 10280 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:51.682112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:48759 (local address 127.0.102.190:46645)
0203 21:07:51.682251 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:51.682256 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:51.682426 (+ 170us) server_negotiation.cc:408] Connection header received
0203 21:07:51.682559 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:51.682562 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:51.682608 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:51.682673 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:51.683221 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:51.683724 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:51.684350 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:51.684542 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:51.686729 (+ 2187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:48759: BlockingRecv error: recv got EOF from 127.0.102.130:48759 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Feb 03 21:07:51 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:51 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:51.854952 10282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:51.850159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:46893)
0203 21:07:51.850465 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:51.850478 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:51.850582 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:51.850806 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:51.850808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:51.850984 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:51.851176 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:51.851182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:51.851902 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:51.851905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:51.852321 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:51.852329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:51.852422 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:51.852932 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:51.852945 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:51.854800 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":110,"threads_started":1}
W20260203 21:07:51.855067 10280 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:51.850321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:46893 (local address 127.0.102.190:46645)
0203 21:07:51.850446 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:51.850449 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:51.850587 (+ 138us) server_negotiation.cc:408] Connection header received
0203 21:07:51.850647 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:51.850650 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:51.850702 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:51.850779 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:51.851291 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:51.851792 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:51.852442 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:51.852586 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:51.854952 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:46893: BlockingRecv error: recv got EOF from 127.0.102.131:46893 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Feb 03 21:07:52 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:52 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:52.540920 10283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:52.535759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:44189)
0203 21:07:52.536036 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:52.536054 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:52.536172 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:52.536503 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:52.536507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:52.536749 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:52.537003 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:52.537014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:52.537824 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:52.537828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:52.538258 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:52.538265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:52.538363 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:52.538899 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:52.538913 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:52.540769 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":95,"threads_started":1}
W20260203 21:07:52.541018 10284 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:52.535876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:44189 (local address 127.0.102.190:46645)
0203 21:07:52.536132 (+ 256us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:52.536137 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:52.536153 (+ 16us) server_negotiation.cc:408] Connection header received
0203 21:07:52.536266 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:52.536269 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:52.536344 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:52.536434 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:52.537153 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:52.537693 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:52.538360 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:52.538522 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:52.540905 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:44189: BlockingRecv error: recv got EOF from 127.0.102.129:44189 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":53,"threads_started":1}
Feb 03 21:07:52 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:52 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:52.692976 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:52.687708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:34707)
0203 21:07:52.687985 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:52.687998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:52.688099 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:52.688428 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:52.688430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:52.688609 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:52.688859 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:52.688866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:52.689721 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:52.689725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:52.690113 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:52.690121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:52.690230 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:52.690751 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:52.690765 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:52.692825 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
W20260203 21:07:52.693053 10284 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:52.687840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:34707 (local address 127.0.102.190:46645)
0203 21:07:52.688004 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:52.688008 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:52.688064 (+ 56us) server_negotiation.cc:408] Connection header received
0203 21:07:52.688223 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:52.688226 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:52.688283 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:52.688383 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:52.689002 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:52.689580 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:52.690239 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:52.690391 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:52.692956 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:34707: BlockingRecv error: recv got EOF from 127.0.102.130:34707 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Feb 03 21:07:52 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:52 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:52.860937 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:52.855864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:37971)
0203 21:07:52.856166 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:52.856182 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:52.856302 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:52.856670 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:52.856673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:52.856863 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:52.857095 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:52.857110 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:52.857812 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:52.857815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:52.858225 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:52.858232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:52.858334 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:52.858836 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:52.858848 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:52.860821 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
W20260203 21:07:52.861065 10284 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:52.856038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:37971 (local address 127.0.102.190:46645)
0203 21:07:52.856165 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:52.856170 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:52.856458 (+ 288us) server_negotiation.cc:408] Connection header received
0203 21:07:52.856496 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:52.856498 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:52.856541 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:52.856607 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:52.857208 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:52.857706 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:52.858325 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:52.858508 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:52.860947 (+ 2439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:37971: BlockingRecv error: recv got EOF from 127.0.102.131:37971 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Feb 03 21:07:53 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:53 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:53.546939 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:53.541861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:56261)
0203 21:07:53.542189 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:53.542204 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:53.542336 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:53.542664 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:53.542667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:53.542846 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:53.543043 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:53.543048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:53.543812 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:53.543816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:53.544276 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:53.544285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:53.544394 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:53.545023 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:53.545038 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:53.546796 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":83,"threads_started":1}
W20260203 21:07:53.547062 10288 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:53.541987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:56261 (local address 127.0.102.190:46645)
0203 21:07:53.542301 (+ 314us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:53.542305 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:53.542320 (+ 15us) server_negotiation.cc:408] Connection header received
0203 21:07:53.542478 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:53.542482 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:53.542534 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:53.542621 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:53.543151 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:53.543699 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:53.544377 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:53.544580 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:53.546934 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:56261: BlockingRecv error: recv got EOF from 127.0.102.129:56261 (error 108)
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":101,"threads_started":1}
Feb 03 21:07:53 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:53 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:53.699229 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:53.694001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:37079)
0203 21:07:53.694269 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:53.694282 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:53.694405 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:53.694714 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:53.694717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:53.694902 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:53.695097 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:53.695103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:53.695869 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:53.695873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:53.696268 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:53.696276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:53.696383 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:53.697042 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:53.697057 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:53.699093 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
W20260203 21:07:53.699376 10288 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:53.694086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:37079 (local address 127.0.102.190:46645)
0203 21:07:53.694282 (+ 196us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:53.694286 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:53.694360 (+ 74us) server_negotiation.cc:408] Connection header received
0203 21:07:53.694542 (+ 182us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:53.694545 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:53.694604 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:53.694687 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:53.695240 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:53.695753 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:53.696381 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:53.696566 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:53.699248 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:37079: BlockingRecv error: recv got EOF from 127.0.102.130:37079 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Feb 03 21:07:53 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:53 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:53.866425 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:53.861669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:39313)
0203 21:07:53.861942 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:53.861957 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:53.862092 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:53.862323 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:53.862325 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:53.862501 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:53.862709 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:53.862714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:53.863520 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:53.863525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:53.863907 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:53.863914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:53.864015 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:53.864576 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:53.864589 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:53.866310 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":70,"threads_started":1}
W20260203 21:07:53.866616 10288 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:53.861723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:39313 (local address 127.0.102.190:46645)
0203 21:07:53.861871 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:53.861876 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:53.862034 (+ 158us) server_negotiation.cc:408] Connection header received
0203 21:07:53.862182 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:53.862185 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:53.862235 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:53.862300 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:53.862847 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:53.863365 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:53.864024 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:53.864165 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:53.866475 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:39313: BlockingRecv error: recv got EOF from 127.0.102.131:39313 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Feb 03 21:07:54 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:54 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:54.555770 10291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:54.547839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:50129)
0203 21:07:54.548101 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:54.548114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:54.548208 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:54.548504 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:54.548507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:54.548736 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:54.548935 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:54.548941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:54.549712 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:54.549715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:54.550107 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:54.550115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:54.550226 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:54.553762 (+ 3536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:54.553781 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:54.555655 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
W20260203 21:07:54.555941 10292 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:54.547939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:50129 (local address 127.0.102.190:46645)
0203 21:07:54.548220 (+ 281us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:54.548225 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:54.548242 (+ 17us) server_negotiation.cc:408] Connection header received
0203 21:07:54.548283 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:54.548285 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:54.548329 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:54.548437 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:54.549048 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:54.549597 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:54.550238 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:54.550447 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:54.555812 (+ 5365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:50129: BlockingRecv error: recv got EOF from 127.0.102.129:50129 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
Feb 03 21:07:54 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:54 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:54.705058 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:54.700142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:53639)
0203 21:07:54.700445 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:54.700459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:54.700570 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:54.700890 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:54.700893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:54.701078 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:54.701297 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:54.701304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:54.702153 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:54.702157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:54.702571 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:54.702579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:54.702696 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:54.703219 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:54.703233 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:54.704940 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":80,"threads_started":1}
W20260203 21:07:54.705235 10292 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:54.700280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:53639 (local address 127.0.102.190:46645)
0203 21:07:54.700434 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:54.700438 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:54.700567 (+ 129us) server_negotiation.cc:408] Connection header received
0203 21:07:54.700695 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:54.700698 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:54.700749 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:54.700824 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:54.701473 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:54.702045 (+ 572us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:54.702730 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:54.702945 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:54.705113 (+ 2168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:53639: BlockingRecv error: recv got EOF from 127.0.102.130:53639 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Feb 03 21:07:54 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:54 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:54.871553 10294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:54.867128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:49251)
0203 21:07:54.867386 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:54.867399 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:54.867504 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:54.867718 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:54.867721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:54.867892 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:54.868084 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:54.868090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:54.868784 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:54.868787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:54.869167 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:54.869175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:54.869264 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:54.869759 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:54.869771 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:54.871438 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
W20260203 21:07:54.871686 10292 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:54.867251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:49251 (local address 127.0.102.190:46645)
0203 21:07:54.867378 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:54.867382 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:54.867465 (+ 83us) server_negotiation.cc:408] Connection header received
0203 21:07:54.867576 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:54.867578 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:54.867624 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:54.867696 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:54.868170 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:54.868692 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:54.869251 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:54.869374 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:54.871575 (+ 2201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:49251: BlockingRecv error: recv got EOF from 127.0.102.131:49251 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Feb 03 21:07:55 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:55 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:55.561834 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:55.556811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:59999)
0203 21:07:55.557135 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:55.557150 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:55.557283 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:55.557650 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:55.557653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:55.557883 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:55.558098 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:55.558106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:55.558903 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:55.558906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:55.559295 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:55.559301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:55.559403 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:55.559917 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:55.559930 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:55.561717 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
W20260203 21:07:55.561969 10296 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:55.556920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:59999 (local address 127.0.102.190:46645)
0203 21:07:55.557171 (+ 251us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:55.557175 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:55.557214 (+ 39us) server_negotiation.cc:408] Connection header received
0203 21:07:55.557418 (+ 204us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:55.557422 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:55.557483 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:55.557580 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:55.558224 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:55.558789 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:55.559398 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:55.559590 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:55.561844 (+ 2254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:59999: BlockingRecv error: recv got EOF from 127.0.102.129:59999 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":61,"threads_started":1}
Feb 03 21:07:55 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:55 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:55.711377 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:55.705942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:38481)
0203 21:07:55.706222 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:55.706236 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:55.706332 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:55.706693 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:55.706696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:55.706877 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:55.707108 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:55.707116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:55.707949 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:55.707952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:55.708355 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:55.708363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:55.708480 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:55.709041 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:55.709055 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:55.711169 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":82,"threads_started":1}
W20260203 21:07:55.711453 10296 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:55.706114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:38481 (local address 127.0.102.190:46645)
0203 21:07:55.706260 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:55.706264 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:55.706285 (+ 21us) server_negotiation.cc:408] Connection header received
0203 21:07:55.706458 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:55.706462 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:55.706516 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:55.706620 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:55.707282 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:55.707821 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:55.708503 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:55.708728 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:55.711343 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:38481: BlockingRecv error: recv got EOF from 127.0.102.130:38481 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Feb 03 21:07:55 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:55 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:55.877099 10298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:55.872396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:44459)
0203 21:07:55.872698 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:55.872711 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:55.872814 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:55.873041 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:55.873043 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:55.873220 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:55.873412 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:55.873417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:55.874130 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:55.874133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:55.874512 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:55.874519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:55.874635 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:55.875128 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:55.875141 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:55.876981 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
W20260203 21:07:55.877300 10296 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:55.872500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:44459 (local address 127.0.102.190:46645)
0203 21:07:55.872703 (+ 203us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:55.872707 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:55.872787 (+ 80us) server_negotiation.cc:408] Connection header received
0203 21:07:55.872889 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:55.872892 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:55.872939 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:55.873019 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:55.873514 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:55.874025 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:55.874617 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:55.874755 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:55.877150 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:44459: BlockingRecv error: recv got EOF from 127.0.102.131:44459 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Feb 03 21:07:56 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:56 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:56.567675 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:56.562641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:41799)
0203 21:07:56.562929 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:56.562946 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:56.563048 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:56.563306 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:56.563309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:56.563486 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:56.563704 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:56.563711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:56.564513 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:56.564516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:56.564982 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:56.564991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:56.565097 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:56.565641 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:56.565655 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:56.567544 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
W20260203 21:07:56.567890 10300 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:56.562735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:41799 (local address 127.0.102.190:46645)
0203 21:07:56.563055 (+ 320us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:56.563059 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:56.563073 (+ 14us) server_negotiation.cc:408] Connection header received
0203 21:07:56.563111 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:56.563113 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:56.563160 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:56.563243 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:56.563863 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:56.564381 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:56.565114 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:56.565268 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:56.567751 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:41799: BlockingRecv error: recv got EOF from 127.0.102.129:41799 (error 108)
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":94,"threads_started":1}
Feb 03 21:07:56 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:56 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:56.717595 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:56.712521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:48637)
0203 21:07:56.712801 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:56.712817 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:56.712952 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:56.713281 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:56.713284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:56.713460 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:56.713701 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:56.713710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:56.714515 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:56.714519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:56.714964 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:56.714973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:56.715080 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:56.715616 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:56.715631 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:56.717466 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":136,"threads_started":1}
W20260203 21:07:56.717808 10300 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:56.712660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:48637 (local address 127.0.102.190:46645)
0203 21:07:56.712786 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:56.712790 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:56.712919 (+ 129us) server_negotiation.cc:408] Connection header received
0203 21:07:56.713069 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:56.713073 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:56.713125 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:56.713214 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:56.713873 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:56.714380 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:56.715093 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:56.715253 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:56.717652 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:48637: BlockingRecv error: recv got EOF from 127.0.102.130:48637 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Feb 03 21:07:56 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:56 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.131@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:56.882854 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:56.877916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.131:48353)
0203 21:07:56.878186 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:56.878199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:56.878305 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:56.878568 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:56.878572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:56.878814 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:56.879033 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:56.879040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:56.879781 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:56.879784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:56.880233 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:56.880242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:56.880340 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:56.880875 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:56.880888 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:56.882737 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":92,"threads_started":1}
W20260203 21:07:56.882995 10300 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:56.878012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:48353 (local address 127.0.102.190:46645)
0203 21:07:56.878170 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:56.878175 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:56.878287 (+ 112us) server_negotiation.cc:408] Connection header received
0203 21:07:56.878381 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:56.878384 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:56.878433 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:56.878502 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:56.879171 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:56.879665 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:56.880342 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:56.880480 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:56.882875 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.131:48353: BlockingRecv error: recv got EOF from 127.0.102.131:48353 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Feb 03 21:07:57 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:57 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.129@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:57.573761 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:57.568589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.129:51019)
0203 21:07:57.568893 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:57.568906 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:57.568990 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:57.569238 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:57.569240 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:57.569426 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:57.569618 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:57.569624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:57.570396 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:57.570400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:57.571067 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:57.571078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:57.571199 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:57.571715 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:57.571735 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:57.573634 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":105,"threads_started":1}
W20260203 21:07:57.573943 10304 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:57.568718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:51019 (local address 127.0.102.190:46645)
0203 21:07:57.568951 (+ 233us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:57.568955 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:57.568969 (+ 14us) server_negotiation.cc:408] Connection header received
0203 21:07:57.569071 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:57.569073 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:57.569118 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:57.569203 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:57.569708 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:57.570248 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:57.571200 (+ 952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:57.571402 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:57.573813 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.129:51019: BlockingRecv error: recv got EOF from 127.0.102.129:51019 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":80,"threads_started":1}
Feb 03 21:07:57 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
Feb 03 21:07:57 dist-test-slave-j889 krb5kdc[9237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.102.130@KRBTEST.COM for oryx/127.0.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260203 21:07:57.723510 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:57.718580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:46645 (local address 127.0.102.130:51255)
0203 21:07:57.718975 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:57.718988 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:57.719093 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:57.719393 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:57.719397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:57.719625 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:57.719855 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:57.719864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:57.720693 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:57.720697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:57.721113 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:57.721121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:57.721214 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:57.721708 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:07:57.721721 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:57.723396 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.102.190:46645: Server oryx/127.0.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":120,"threads_started":1}
W20260203 21:07:57.723655 10304 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:57.718695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:51255 (local address 127.0.102.190:46645)
0203 21:07:57.718848 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:57.718853 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:57.719055 (+ 202us) server_negotiation.cc:408] Connection header received
0203 21:07:57.719184 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:57.719188 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:57.719244 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:57.719341 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:57.719958 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:57.720553 (+ 595us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:57.721206 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:57.721345 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:57.723527 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.102.130:51255: BlockingRecv error: recv got EOF from 127.0.102.130:51255 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260203 21:07:57.800951 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 9829
I20260203 21:07:57.805999 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 9981
I20260203 21:07:57.810767 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 10113
I20260203 21:07:57.815300 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 9759
2026-02-03T21:07:57Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18813 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-02-03T21:07:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:07:57Z Disabled control of system clock
I20260203 21:07:57.843696 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:43263
--webserver_interface=127.0.102.190
--webserver_port=0
--builtin_ntp_servers=127.0.102.148:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:43263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260203 21:07:57.953150 10309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:57.953397 10309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:57.953439 10309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:57.957643 10309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:07:57.957726 10309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:57.957747 10309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:57.957784 10309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:07:57.957813 10309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:07:57.961794 10309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:43263
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:43263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:57.962888 10309 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:57.963990 10309 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:57.969518 10317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:57.969725 10309 server_base.cc:1060] running on GCE node
W20260203 21:07:57.969516 10314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:57.969501 10315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:57.970353 10309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:57.971231 10309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:57.972374 10309 hybrid_clock.cc:648] HybridClock initialized: now 1770152877972360 us; error 34 us; skew 500 ppm
I20260203 21:07:57.974066 10309 webserver.cc:492] Webserver started at http://127.0.102.190:43227/ using document root <none> and password file <none>
I20260203 21:07:57.974612 10309 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:57.974664 10309 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:57.974814 10309 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:57.976485 10309 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "ace6c8d5831d4ab8be4a12836690c20d"
format_stamp: "Formatted at 2026-02-03 21:07:57 on dist-test-slave-j889"
server_key: "4909c08dae1afcc9398e7756a300841c"
server_key_iv: "495c41a7d618075967748a9e3f4c2c92"
server_key_version: "encryptionkey@0"
I20260203 21:07:57.976982 10309 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "ace6c8d5831d4ab8be4a12836690c20d"
format_stamp: "Formatted at 2026-02-03 21:07:57 on dist-test-slave-j889"
server_key: "4909c08dae1afcc9398e7756a300841c"
server_key_iv: "495c41a7d618075967748a9e3f4c2c92"
server_key_version: "encryptionkey@0"
I20260203 21:07:57.980352 10309 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260203 21:07:57.982735 10323 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:57.983943 10309 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:57.984052 10309 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "ace6c8d5831d4ab8be4a12836690c20d"
format_stamp: "Formatted at 2026-02-03 21:07:57 on dist-test-slave-j889"
server_key: "4909c08dae1afcc9398e7756a300841c"
server_key_iv: "495c41a7d618075967748a9e3f4c2c92"
server_key_version: "encryptionkey@0"
I20260203 21:07:57.984144 10309 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:58.003931 10309 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:58.004598 10309 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:58.004820 10309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:58.011855 10309 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:43263
I20260203 21:07:58.011850 10375 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:43263 every 8 connection(s)
I20260203 21:07:58.012910 10309 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:07:58.015758 10376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:58.018893 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 10309
I20260203 21:07:58.018999 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:07:58.019227 410 external_mini_cluster.cc:1468] Setting key 6323eaa78430d6e313a45d7c892aae36
I20260203 21:07:58.021574 10376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d: Bootstrap starting.
I20260203 21:07:58.023903 10376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:58.024807 10376 log.cc:826] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:58.026767 10376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d: No bootstrap required, opened a new log
I20260203 21:07:58.026947 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:58.020514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41686 (local address 127.0.102.190:43263)
0203 21:07:58.021141 (+ 627us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:58.021150 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:58.021184 (+ 34us) server_negotiation.cc:408] Connection header received
0203 21:07:58.021702 (+ 518us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:58.021727 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:58.022099 (+ 372us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:58.022374 (+ 275us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:58.022976 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:58.023725 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:58.024600 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:58.024897 (+ 297us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:58.025396 (+ 499us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:58.025416 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:58.025427 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:58.025789 (+ 362us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:58.025832 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:58.025846 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:58.025950 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:58.026105 (+ 155us) server_negotiation.cc:300] Negotiation successful
0203 21:07:58.026335 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":147,"threads_started":1}
I20260203 21:07:58.029721 10376 raft_consensus.cc:359] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace6c8d5831d4ab8be4a12836690c20d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43263 } }
I20260203 21:07:58.029925 10376 raft_consensus.cc:385] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:58.029961 10376 raft_consensus.cc:740] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace6c8d5831d4ab8be4a12836690c20d, State: Initialized, Role: FOLLOWER
I20260203 21:07:58.030385 10376 consensus_queue.cc:260] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace6c8d5831d4ab8be4a12836690c20d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43263 } }
I20260203 21:07:58.030520 10376 raft_consensus.cc:399] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:07:58.030579 10376 raft_consensus.cc:493] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:07:58.030666 10376 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:58.031538 10376 raft_consensus.cc:515] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace6c8d5831d4ab8be4a12836690c20d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43263 } }
I20260203 21:07:58.031883 10376 leader_election.cc:304] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ace6c8d5831d4ab8be4a12836690c20d; no voters:
I20260203 21:07:58.032191 10376 leader_election.cc:290] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:07:58.032332 10381 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:58.032579 10381 raft_consensus.cc:697] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [term 1 LEADER]: Becoming Leader. State: Replica: ace6c8d5831d4ab8be4a12836690c20d, State: Running, Role: LEADER
I20260203 21:07:58.032882 10381 consensus_queue.cc:237] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace6c8d5831d4ab8be4a12836690c20d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43263 } }
I20260203 21:07:58.033345 10376 sys_catalog.cc:565] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:07:58.034340 10382 sys_catalog.cc:455] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ace6c8d5831d4ab8be4a12836690c20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace6c8d5831d4ab8be4a12836690c20d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43263 } } }
I20260203 21:07:58.034463 10382 sys_catalog.cc:458] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:58.034425 10383 sys_catalog.cc:455] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ace6c8d5831d4ab8be4a12836690c20d. Latest consensus state: current_term: 1 leader_uuid: "ace6c8d5831d4ab8be4a12836690c20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ace6c8d5831d4ab8be4a12836690c20d" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 43263 } } }
I20260203 21:07:58.034516 10383 sys_catalog.cc:458] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:58.035046 10390 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:07:58.038079 10390 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:07:58.043489 10390 catalog_manager.cc:1357] Generated new cluster ID: 41811405ecbd4608a25642d5c861b448
I20260203 21:07:58.043577 10390 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:07:58.053188 10390 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:07:58.054106 10390 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:07:58.061288 10390 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ace6c8d5831d4ab8be4a12836690c20d: Generated new TSK 0
I20260203 21:07:58.062058 10390 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260203 21:07:58.073510 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:43263
--builtin_ntp_servers=127.0.102.148:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260203 21:07:58.187834 10400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:58.188088 10400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:58.188163 10400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:58.191684 10400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:58.191762 10400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:58.191860 10400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:07:58.195816 10400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:43263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:58.197012 10400 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:58.198179 10400 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:58.204799 10406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:58.204790 10405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:58.204850 10408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:58.205252 10400 server_base.cc:1060] running on GCE node
I20260203 21:07:58.205628 10400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:58.206194 10400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:58.207350 10400 hybrid_clock.cc:648] HybridClock initialized: now 1770152878207330 us; error 38 us; skew 500 ppm
I20260203 21:07:58.209283 10400 webserver.cc:492] Webserver started at http://127.0.102.129:44963/ using document root <none> and password file <none>
I20260203 21:07:58.209846 10400 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:58.209903 10400 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:58.210090 10400 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:58.211772 10400 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "037c269054d245c9ab6cc32da65a2596"
format_stamp: "Formatted at 2026-02-03 21:07:58 on dist-test-slave-j889"
server_key: "f4dc7ac617ad6e3822594bd2fadf818e"
server_key_iv: "0bed150344f4f14bf3127759b618bae1"
server_key_version: "encryptionkey@0"
I20260203 21:07:58.212248 10400 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "037c269054d245c9ab6cc32da65a2596"
format_stamp: "Formatted at 2026-02-03 21:07:58 on dist-test-slave-j889"
server_key: "f4dc7ac617ad6e3822594bd2fadf818e"
server_key_iv: "0bed150344f4f14bf3127759b618bae1"
server_key_version: "encryptionkey@0"
I20260203 21:07:58.215667 10400 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260203 21:07:58.218046 10414 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:58.219197 10400 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260203 21:07:58.219314 10400 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "037c269054d245c9ab6cc32da65a2596"
format_stamp: "Formatted at 2026-02-03 21:07:58 on dist-test-slave-j889"
server_key: "f4dc7ac617ad6e3822594bd2fadf818e"
server_key_iv: "0bed150344f4f14bf3127759b618bae1"
server_key_version: "encryptionkey@0"
I20260203 21:07:58.219412 10400 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:58.237326 10400 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:58.238010 10400 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:58.238190 10400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:58.238744 10400 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:58.239763 10400 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:58.239813 10400 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:58.239866 10400 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:58.239892 10400 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:58.248708 10400 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:38439
I20260203 21:07:58.248739 10527 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:38439 every 8 connection(s)
I20260203 21:07:58.249647 10400 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:07:58.257731 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:58.251330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:54271 (local address 127.0.102.190:43263)
0203 21:07:58.251456 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:58.251460 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:58.252402 (+ 942us) server_negotiation.cc:408] Connection header received
0203 21:07:58.253281 (+ 879us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:58.253285 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:58.253349 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:58.253424 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:58.254441 (+ 1017us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:58.254958 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:58.255629 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:58.255816 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:58.257215 (+ 1399us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:58.257236 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:58.257238 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:58.257252 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:58.257273 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:58.257280 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:58.257341 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:58.257501 (+ 160us) server_negotiation.cc:300] Negotiation successful
0203 21:07:58.257574 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260203 21:07:58.258556 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:58.251586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:43263 (local address 127.0.102.129:54271)
0203 21:07:58.252265 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:58.252299 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:58.253073 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:58.253598 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:58.253607 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:58.253690 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:58.254290 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:58.254304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:58.255088 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:58.255091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:58.255506 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:58.255514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:58.255752 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:58.256978 (+ 1226us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0203 21:07:58.256980 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:58.257016 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0203 21:07:58.257079 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:58.257353 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:58.257361 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:58.257366 (+ 5us) client_negotiation.cc:769] Sending connection context
0203 21:07:58.257501 (+ 135us) client_negotiation.cc:240] Negotiation successful
0203 21:07:58.257763 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":107,"threads_started":1}
I20260203 21:07:58.259806 10528 heartbeater.cc:344] Connected to a master server at 127.0.102.190:43263
I20260203 21:07:58.259944 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 10400
I20260203 21:07:58.260077 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:07:58.260095 10528 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:58.260366 410 external_mini_cluster.cc:1468] Setting key def650ec3d874412087361f8d0f5aba4
I20260203 21:07:58.260810 10528 heartbeater.cc:507] Master 127.0.102.190:43263 requested a full tablet report, sending...
I20260203 21:07:58.262230 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:43263
--builtin_ntp_servers=127.0.102.148:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260203 21:07:58.262257 10340 ts_manager.cc:194] Registered new tserver with Master: 037c269054d245c9ab6cc32da65a2596 (127.0.102.129:38439)
I20260203 21:07:58.263435 10340 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.102.129:54271
W20260203 21:07:58.370355 10531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:58.370581 10531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:58.370633 10531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:58.374035 10531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:58.374109 10531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:58.374182 10531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:07:58.378211 10531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:43263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:58.379328 10531 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:58.380436 10531 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:58.387343 10539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:58.387310 10536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:58.387570 10531 server_base.cc:1060] running on GCE node
W20260203 21:07:58.387802 10537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:58.388180 10531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:58.388836 10531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:58.390040 10531 hybrid_clock.cc:648] HybridClock initialized: now 1770152878390022 us; error 31 us; skew 500 ppm
I20260203 21:07:58.391968 10531 webserver.cc:492] Webserver started at http://127.0.102.130:40969/ using document root <none> and password file <none>
I20260203 21:07:58.392551 10531 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:58.392614 10531 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:58.392844 10531 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:58.394807 10531 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "9a6efb5518ef463abd456c3f2b84382e"
format_stamp: "Formatted at 2026-02-03 21:07:58 on dist-test-slave-j889"
server_key: "72ece8722e61376f2eb04f602be3463f"
server_key_iv: "89ebd29eab6e1469eb428a3d844697a3"
server_key_version: "encryptionkey@0"
I20260203 21:07:58.395335 10531 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "9a6efb5518ef463abd456c3f2b84382e"
format_stamp: "Formatted at 2026-02-03 21:07:58 on dist-test-slave-j889"
server_key: "72ece8722e61376f2eb04f602be3463f"
server_key_iv: "89ebd29eab6e1469eb428a3d844697a3"
server_key_version: "encryptionkey@0"
I20260203 21:07:58.398934 10531 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260203 21:07:58.401259 10545 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:58.402287 10531 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:07:58.402398 10531 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "9a6efb5518ef463abd456c3f2b84382e"
format_stamp: "Formatted at 2026-02-03 21:07:58 on dist-test-slave-j889"
server_key: "72ece8722e61376f2eb04f602be3463f"
server_key_iv: "89ebd29eab6e1469eb428a3d844697a3"
server_key_version: "encryptionkey@0"
I20260203 21:07:58.402493 10531 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:58.421085 10531 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:58.421759 10531 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:58.421934 10531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:58.422504 10531 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:58.423514 10531 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:58.423563 10531 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:58.423615 10531 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:58.423640 10531 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:58.433733 10531 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:44513
I20260203 21:07:58.433769 10658 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:44513 every 8 connection(s)
I20260203 21:07:58.435103 10531 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:07:58.438479 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 10531
I20260203 21:07:58.438609 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:07:58.438861 410 external_mini_cluster.cc:1468] Setting key 58c6c258044b1d45049a654a01c96c15
I20260203 21:07:58.440913 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:43263
--builtin_ntp_servers=127.0.102.148:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260203 21:07:58.445623 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:58.437351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:38083 (local address 127.0.102.190:43263)
0203 21:07:58.437546 (+ 195us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:58.437551 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:58.438745 (+ 1194us) server_negotiation.cc:408] Connection header received
0203 21:07:58.439866 (+ 1121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:58.439870 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:58.439950 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:58.440089 (+ 139us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:58.441404 (+ 1315us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:58.442109 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:58.443055 (+ 946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:58.443228 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:58.444929 (+ 1701us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:58.444973 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:58.444976 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:58.444990 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:58.445018 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:58.445030 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:58.445104 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:58.445428 (+ 324us) server_negotiation.cc:300] Negotiation successful
0203 21:07:58.445497 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260203 21:07:58.446722 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:58.437688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:43263 (local address 127.0.102.130:38083)
0203 21:07:58.438568 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:58.438618 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:58.439600 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:58.440408 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:58.440421 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:58.440511 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:58.441253 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:58.441268 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:58.442294 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:58.442299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:58.442916 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:58.442926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:58.443183 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:58.444578 (+ 1395us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0203 21:07:58.444581 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:58.444658 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0203 21:07:58.444747 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:58.445211 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:58.445226 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:58.445233 (+ 7us) client_negotiation.cc:769] Sending connection context
0203 21:07:58.445426 (+ 193us) client_negotiation.cc:240] Negotiation successful
0203 21:07:58.445732 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":130,"threads_started":1}
I20260203 21:07:58.448153 10659 heartbeater.cc:344] Connected to a master server at 127.0.102.190:43263
I20260203 21:07:58.448434 10659 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:58.449115 10659 heartbeater.cc:507] Master 127.0.102.190:43263 requested a full tablet report, sending...
I20260203 21:07:58.450390 10340 ts_manager.cc:194] Registered new tserver with Master: 9a6efb5518ef463abd456c3f2b84382e (127.0.102.130:44513)
I20260203 21:07:58.450961 10340 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.102.130:38083
W20260203 21:07:58.553424 10662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:58.553674 10662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:58.553732 10662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:58.557193 10662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:58.557265 10662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:58.557343 10662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:07:58.561363 10662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:43263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:58.562495 10662 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:58.563619 10662 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:58.570170 10670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:58.570192 10667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:58.570336 10662 server_base.cc:1060] running on GCE node
W20260203 21:07:58.570477 10668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:58.570907 10662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:58.571549 10662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:58.572715 10662 hybrid_clock.cc:648] HybridClock initialized: now 1770152878572693 us; error 42 us; skew 500 ppm
I20260203 21:07:58.574609 10662 webserver.cc:492] Webserver started at http://127.0.102.131:41443/ using document root <none> and password file <none>
I20260203 21:07:58.575225 10662 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:58.575286 10662 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:58.575471 10662 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:58.577320 10662 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "9a45254ba66a48e7bc8b4b5cee0683ed"
format_stamp: "Formatted at 2026-02-03 21:07:58 on dist-test-slave-j889"
server_key: "8a311c584c2f7e694dfd1eed642270ae"
server_key_iv: "4dcb23c675f0e958e07b9b94542ad76b"
server_key_version: "encryptionkey@0"
I20260203 21:07:58.577919 10662 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "9a45254ba66a48e7bc8b4b5cee0683ed"
format_stamp: "Formatted at 2026-02-03 21:07:58 on dist-test-slave-j889"
server_key: "8a311c584c2f7e694dfd1eed642270ae"
server_key_iv: "4dcb23c675f0e958e07b9b94542ad76b"
server_key_version: "encryptionkey@0"
I20260203 21:07:58.581705 10662 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260203 21:07:58.584087 10676 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:58.585296 10662 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:07:58.585412 10662 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "9a45254ba66a48e7bc8b4b5cee0683ed"
format_stamp: "Formatted at 2026-02-03 21:07:58 on dist-test-slave-j889"
server_key: "8a311c584c2f7e694dfd1eed642270ae"
server_key_iv: "4dcb23c675f0e958e07b9b94542ad76b"
server_key_version: "encryptionkey@0"
I20260203 21:07:58.585510 10662 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:58.612843 10662 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:58.613758 10662 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:58.613984 10662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:58.614683 10662 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:58.615729 10662 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:58.615780 10662 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:58.615834 10662 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:58.615862 10662 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:58.624895 10662 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:42561
I20260203 21:07:58.624925 10789 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:42561 every 8 connection(s)
I20260203 21:07:58.625922 10662 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
I20260203 21:07:58.627328 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 10662
I20260203 21:07:58.627454 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:07:58.627736 410 external_mini_cluster.cc:1468] Setting key a01b36726605544367d734c74e085a84
I20260203 21:07:58.634431 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:58.627893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:39437 (local address 127.0.102.190:43263)
0203 21:07:58.628036 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:58.628041 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:58.629305 (+ 1264us) server_negotiation.cc:408] Connection header received
0203 21:07:58.630262 (+ 957us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:58.630265 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:58.630321 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:58.630395 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:58.631411 (+ 1016us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:58.631896 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:07:58.632517 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:07:58.632720 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:58.633931 (+ 1211us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:58.633951 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:58.633954 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:58.633967 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:58.633988 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:58.633996 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:58.634055 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:58.634219 (+ 164us) server_negotiation.cc:300] Negotiation successful
0203 21:07:58.634280 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260203 21:07:58.635169 10792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:58.628293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:43263 (local address 127.0.102.131:39437)
0203 21:07:58.629174 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:58.629219 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:58.630061 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:58.630571 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:58.630580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:58.630668 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:58.631270 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:58.631282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:58.632015 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:07:58.632018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:07:58.632407 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:07:58.632414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:07:58.632659 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:07:58.633704 (+ 1045us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0203 21:07:58.633705 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:58.633740 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0203 21:07:58.633797 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:58.634069 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:58.634078 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:58.634083 (+ 5us) client_negotiation.cc:769] Sending connection context
0203 21:07:58.634212 (+ 129us) client_negotiation.cc:240] Negotiation successful
0203 21:07:58.634451 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":149,"threads_started":1}
I20260203 21:07:58.636144 10790 heartbeater.cc:344] Connected to a master server at 127.0.102.190:43263
I20260203 21:07:58.636376 10790 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:58.636968 10790 heartbeater.cc:507] Master 127.0.102.190:43263 requested a full tablet report, sending...
I20260203 21:07:58.637974 10340 ts_manager.cc:194] Registered new tserver with Master: 9a45254ba66a48e7bc8b4b5cee0683ed (127.0.102.131:42561)
I20260203 21:07:58.638551 10340 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.102.131:39437
I20260203 21:07:58.641575 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260203 21:07:58.644187 10379 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:58.643365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41702 (local address 127.0.102.190:43263)
0203 21:07:58.643503 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:58.643507 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:58.643623 (+ 116us) server_negotiation.cc:408] Connection header received
0203 21:07:58.643740 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:58.643743 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:58.643786 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:58.643849 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:58.644054 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41702: BlockingRecv error: recv got EOF from 127.0.0.1:41702 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260203 21:07:58.644593 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 10400
I20260203 21:07:58.650616 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 10531
I20260203 21:07:58.655578 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 10662
I20260203 21:07:58.660388 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 10309
2026-02-03T21:07:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (843 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-02-03T21:07:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:07:58Z Disabled control of system clock
I20260203 21:07:58.687475 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:45105
--webserver_interface=127.0.102.190
--webserver_port=0
--builtin_ntp_servers=127.0.102.148:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:45105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260203 21:07:58.797524 10801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:58.797775 10801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:58.797823 10801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:58.801282 10801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:07:58.801371 10801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:58.801391 10801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:58.801414 10801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:07:58.801431 10801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:07:58.805864 10801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:45105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:45105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:58.807050 10801 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:58.808198 10801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:58.814374 10806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:58.814306 10807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:58.814646 10809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:58.814719 10801 server_base.cc:1060] running on GCE node
I20260203 21:07:58.815323 10801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:58.816375 10801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:58.817610 10801 hybrid_clock.cc:648] HybridClock initialized: now 1770152878817547 us; error 81 us; skew 500 ppm
I20260203 21:07:58.819769 10801 webserver.cc:492] Webserver started at http://127.0.102.190:41235/ using document root <none> and password file <none>
I20260203 21:07:58.820535 10801 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:58.820614 10801 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:58.820863 10801 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:58.822691 10801 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "b4cfaafde3a3478083f730f4ffa22edd"
format_stamp: "Formatted at 2026-02-03 21:07:58 on dist-test-slave-j889"
server_key: "f5bbb2555d115cd6090c9d00aa56b941"
server_key_iv: "027eb2f37c789334ce7fe0ad06a6809d"
server_key_version: "encryptionkey@0"
I20260203 21:07:58.823226 10801 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "b4cfaafde3a3478083f730f4ffa22edd"
format_stamp: "Formatted at 2026-02-03 21:07:58 on dist-test-slave-j889"
server_key: "f5bbb2555d115cd6090c9d00aa56b941"
server_key_iv: "027eb2f37c789334ce7fe0ad06a6809d"
server_key_version: "encryptionkey@0"
I20260203 21:07:58.827171 10801 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260203 21:07:58.829826 10815 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:58.830976 10801 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:07:58.831115 10801 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "b4cfaafde3a3478083f730f4ffa22edd"
format_stamp: "Formatted at 2026-02-03 21:07:58 on dist-test-slave-j889"
server_key: "f5bbb2555d115cd6090c9d00aa56b941"
server_key_iv: "027eb2f37c789334ce7fe0ad06a6809d"
server_key_version: "encryptionkey@0"
I20260203 21:07:58.831218 10801 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:58.836287 10801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:58.836931 10801 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:58.837090 10801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:58.843894 10801 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:45105
I20260203 21:07:58.843894 10867 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:45105 every 8 connection(s)
I20260203 21:07:58.845021 10801 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:07:58.847833 10868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:07:58.852841 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 10801
I20260203 21:07:58.852968 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:07:58.853235 410 external_mini_cluster.cc:1468] Setting key df91987f773b76fc2326b72a807c936b
I20260203 21:07:58.853818 10868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd: Bootstrap starting.
I20260203 21:07:58.856170 10868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd: Neither blocks nor log segments found. Creating new log.
I20260203 21:07:58.856984 10868 log.cc:826] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd: Log is configured to *not* fsync() on all Append() calls
I20260203 21:07:58.858601 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:58.854499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42820 (local address 127.0.102.190:45105)
0203 21:07:58.855226 (+ 727us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:58.855235 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:58.855272 (+ 37us) server_negotiation.cc:408] Connection header received
0203 21:07:58.855828 (+ 556us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:58.855845 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:58.856201 (+ 356us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:58.856729 (+ 528us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:58.857490 (+ 761us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:58.857495 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:58.857504 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:58.857515 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:58.857558 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:58.857571 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:58.857679 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:58.857806 (+ 127us) server_negotiation.cc:300] Negotiation successful
0203 21:07:58.858030 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":154,"threads_started":1}
I20260203 21:07:58.858958 10868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd: No bootstrap required, opened a new log
I20260203 21:07:58.861581 10868 raft_consensus.cc:359] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4cfaafde3a3478083f730f4ffa22edd" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 45105 } }
I20260203 21:07:58.861774 10868 raft_consensus.cc:385] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:07:58.861835 10868 raft_consensus.cc:740] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4cfaafde3a3478083f730f4ffa22edd, State: Initialized, Role: FOLLOWER
I20260203 21:07:58.862253 10868 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4cfaafde3a3478083f730f4ffa22edd" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 45105 } }
I20260203 21:07:58.862371 10868 raft_consensus.cc:399] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:07:58.862426 10868 raft_consensus.cc:493] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:07:58.862511 10868 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:07:58.863396 10868 raft_consensus.cc:515] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4cfaafde3a3478083f730f4ffa22edd" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 45105 } }
I20260203 21:07:58.863729 10868 leader_election.cc:304] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4cfaafde3a3478083f730f4ffa22edd; no voters:
I20260203 21:07:58.864023 10868 leader_election.cc:290] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:07:58.864164 10873 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:07:58.864396 10873 raft_consensus.cc:697] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [term 1 LEADER]: Becoming Leader. State: Replica: b4cfaafde3a3478083f730f4ffa22edd, State: Running, Role: LEADER
I20260203 21:07:58.864732 10873 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4cfaafde3a3478083f730f4ffa22edd" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 45105 } }
I20260203 21:07:58.865283 10868 sys_catalog.cc:565] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:07:58.866720 10875 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4cfaafde3a3478083f730f4ffa22edd. Latest consensus state: current_term: 1 leader_uuid: "b4cfaafde3a3478083f730f4ffa22edd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4cfaafde3a3478083f730f4ffa22edd" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 45105 } } }
I20260203 21:07:58.866746 10874 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4cfaafde3a3478083f730f4ffa22edd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4cfaafde3a3478083f730f4ffa22edd" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 45105 } } }
I20260203 21:07:58.866839 10874 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [sys.catalog]: This master's current role is: LEADER
I20260203 21:07:58.866838 10875 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd [sys.catalog]: This master's current role is: LEADER
W20260203 21:07:58.870133 10888 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260203 21:07:58.870247 10888 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260203 21:07:58.870343 10884 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:07:58.871135 10884 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:07:58.876194 10884 catalog_manager.cc:1357] Generated new cluster ID: d17d2fc4096d4e20ad0930c859cb0601
I20260203 21:07:58.876281 10884 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:07:58.890544 10884 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:07:58.890887 10884 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:07:58.900342 10884 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b4cfaafde3a3478083f730f4ffa22edd: Generated new TSK 0
I20260203 21:07:58.901254 10884 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260203 21:07:58.905750 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:45105
--builtin_ntp_servers=127.0.102.148:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260203 21:07:59.042177 10892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:59.042412 10892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:59.042466 10892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:59.045920 10892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:59.046000 10892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:59.046082 10892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:07:59.050066 10892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:45105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:59.051196 10892 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:59.052371 10892 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:59.059474 10900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:59.059504 10898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:59.059477 10897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:59.059904 10892 server_base.cc:1060] running on GCE node
I20260203 21:07:59.060395 10892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:59.061196 10892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:59.062381 10892 hybrid_clock.cc:648] HybridClock initialized: now 1770152879062355 us; error 44 us; skew 500 ppm
I20260203 21:07:59.064492 10892 webserver.cc:492] Webserver started at http://127.0.102.129:32863/ using document root <none> and password file <none>
I20260203 21:07:59.065207 10892 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:59.065279 10892 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:59.065486 10892 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:59.067790 10892 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "9487ee8255ad4e07a8918b49377a1329"
format_stamp: "Formatted at 2026-02-03 21:07:59 on dist-test-slave-j889"
server_key: "e306d938f79f1b9248e3cc40c0a12bcd"
server_key_iv: "2e66b166c37f77dd4dd0940c09f30dff"
server_key_version: "encryptionkey@0"
I20260203 21:07:59.068436 10892 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "9487ee8255ad4e07a8918b49377a1329"
format_stamp: "Formatted at 2026-02-03 21:07:59 on dist-test-slave-j889"
server_key: "e306d938f79f1b9248e3cc40c0a12bcd"
server_key_iv: "2e66b166c37f77dd4dd0940c09f30dff"
server_key_version: "encryptionkey@0"
I20260203 21:07:59.072698 10892 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260203 21:07:59.075394 10906 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:59.076531 10892 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260203 21:07:59.076665 10892 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "9487ee8255ad4e07a8918b49377a1329"
format_stamp: "Formatted at 2026-02-03 21:07:59 on dist-test-slave-j889"
server_key: "e306d938f79f1b9248e3cc40c0a12bcd"
server_key_iv: "2e66b166c37f77dd4dd0940c09f30dff"
server_key_version: "encryptionkey@0"
I20260203 21:07:59.076772 10892 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:59.081704 10892 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:59.082314 10892 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:59.082468 10892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:59.083000 10892 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:59.083949 10892 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:59.083994 10892 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:59.084034 10892 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:59.084062 10892 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:59.092852 10892 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:41581
I20260203 21:07:59.092885 11019 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:41581 every 8 connection(s)
I20260203 21:07:59.093801 10892 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:07:59.098902 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:59.095548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:45665 (local address 127.0.102.190:45105)
0203 21:07:59.095725 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:59.095729 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:59.096603 (+ 874us) server_negotiation.cc:408] Connection header received
0203 21:07:59.097463 (+ 860us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:59.097466 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:59.097506 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:59.097581 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:59.098546 (+ 965us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:59.098550 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:59.098552 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:59.098563 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:59.098581 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:59.098588 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:59.098662 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:59.098762 (+ 100us) server_negotiation.cc:300] Negotiation successful
0203 21:07:59.098794 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260203 21:07:59.099648 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:59.095803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:45105 (local address 127.0.102.129:45665)
0203 21:07:59.096470 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:59.096505 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:59.097286 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:59.097729 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:59.097737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:59.097818 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:59.098325 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0203 21:07:59.098327 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:59.098360 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0203 21:07:59.098412 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:59.098653 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:59.098658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:59.098661 (+ 3us) client_negotiation.cc:769] Sending connection context
0203 21:07:59.098815 (+ 154us) client_negotiation.cc:240] Negotiation successful
0203 21:07:59.099026 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":98,"threads_started":1}
I20260203 21:07:59.100867 11020 heartbeater.cc:344] Connected to a master server at 127.0.102.190:45105
I20260203 21:07:59.101125 11020 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:59.101526 11020 heartbeater.cc:507] Master 127.0.102.190:45105 requested a full tablet report, sending...
I20260203 21:07:59.101998 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 10892
I20260203 21:07:59.102141 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:07:59.102419 410 external_mini_cluster.cc:1468] Setting key c92cf312ddb531b862c9e66aea8b01e7
I20260203 21:07:59.103117 10832 ts_manager.cc:194] Registered new tserver with Master: 9487ee8255ad4e07a8918b49377a1329 (127.0.102.129:41581)
I20260203 21:07:59.104372 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:45105
--builtin_ntp_servers=127.0.102.148:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260203 21:07:59.211248 11023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:59.211490 11023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:59.211563 11023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:59.214912 11023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:59.214977 11023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:59.215085 11023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:07:59.219038 11023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:45105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:59.220345 11023 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:59.221560 11023 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:59.228437 11028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:59.228819 11031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:59.228988 11023 server_base.cc:1060] running on GCE node
W20260203 21:07:59.228988 11029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:59.229475 11023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:59.230073 11023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:59.231244 11023 hybrid_clock.cc:648] HybridClock initialized: now 1770152879231229 us; error 27 us; skew 500 ppm
I20260203 21:07:59.233276 11023 webserver.cc:492] Webserver started at http://127.0.102.130:35093/ using document root <none> and password file <none>
I20260203 21:07:59.233881 11023 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:59.233939 11023 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:59.234138 11023 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:59.235841 11023 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "231476aadd0b4bbeb9470c387fdab6eb"
format_stamp: "Formatted at 2026-02-03 21:07:59 on dist-test-slave-j889"
server_key: "659cb5c9603836b8ea4954f26335f2df"
server_key_iv: "75cd8f286410f2e0052fd632cc79bd6c"
server_key_version: "encryptionkey@0"
I20260203 21:07:59.236305 11023 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "231476aadd0b4bbeb9470c387fdab6eb"
format_stamp: "Formatted at 2026-02-03 21:07:59 on dist-test-slave-j889"
server_key: "659cb5c9603836b8ea4954f26335f2df"
server_key_iv: "75cd8f286410f2e0052fd632cc79bd6c"
server_key_version: "encryptionkey@0"
I20260203 21:07:59.239789 11023 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260203 21:07:59.242238 11037 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:59.243355 11023 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260203 21:07:59.243458 11023 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "231476aadd0b4bbeb9470c387fdab6eb"
format_stamp: "Formatted at 2026-02-03 21:07:59 on dist-test-slave-j889"
server_key: "659cb5c9603836b8ea4954f26335f2df"
server_key_iv: "75cd8f286410f2e0052fd632cc79bd6c"
server_key_version: "encryptionkey@0"
I20260203 21:07:59.243561 11023 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:59.248365 11023 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:59.248994 11023 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:59.249152 11023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:59.249660 11023 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:59.250589 11023 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:59.250635 11023 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:59.250671 11023 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:59.250697 11023 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:59.259429 11023 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:33315
I20260203 21:07:59.259513 11150 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:33315 every 8 connection(s)
I20260203 21:07:59.260416 11023 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:07:59.266328 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:59.262392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:45035 (local address 127.0.102.190:45105)
0203 21:07:59.262569 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:59.262573 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:59.263529 (+ 956us) server_negotiation.cc:408] Connection header received
0203 21:07:59.264422 (+ 893us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:59.264427 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:59.264488 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:59.264572 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:59.265750 (+ 1178us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:59.265753 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:59.265755 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:59.265768 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:59.265825 (+ 57us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:59.265835 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:59.265928 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:59.266095 (+ 167us) server_negotiation.cc:300] Negotiation successful
0203 21:07:59.266157 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260203 21:07:59.267136 11153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:59.262642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:45105 (local address 127.0.102.130:45035)
0203 21:07:59.263369 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:59.263406 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:59.264165 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:59.264771 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:59.264781 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:59.264871 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:59.265495 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0203 21:07:59.265497 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:59.265534 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0203 21:07:59.265596 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:59.265914 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:59.265923 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:59.265928 (+ 5us) client_negotiation.cc:769] Sending connection context
0203 21:07:59.266167 (+ 239us) client_negotiation.cc:240] Negotiation successful
0203 21:07:59.266416 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":123,"threads_started":1}
I20260203 21:07:59.268435 11151 heartbeater.cc:344] Connected to a master server at 127.0.102.190:45105
I20260203 21:07:59.268739 11151 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:59.269205 11151 heartbeater.cc:507] Master 127.0.102.190:45105 requested a full tablet report, sending...
I20260203 21:07:59.269821 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 11023
I20260203 21:07:59.269924 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:07:59.270152 410 external_mini_cluster.cc:1468] Setting key 4fb69fe34a121c92c0637ed8491fd8f5
I20260203 21:07:59.270473 10832 ts_manager.cc:194] Registered new tserver with Master: 231476aadd0b4bbeb9470c387fdab6eb (127.0.102.130:33315)
I20260203 21:07:59.271914 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:45105
--builtin_ntp_servers=127.0.102.148:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260203 21:07:59.382630 11154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:07:59.382899 11154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:07:59.382956 11154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:07:59.386518 11154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:07:59.386598 11154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:07:59.386682 11154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:07:59.390841 11154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:45105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:07:59.391985 11154 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:07:59.393154 11154 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:07:59.399745 11159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:59.399719 11162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:07:59.399700 11160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:07:59.399959 11154 server_base.cc:1060] running on GCE node
I20260203 21:07:59.400516 11154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:07:59.401134 11154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:07:59.402311 11154 hybrid_clock.cc:648] HybridClock initialized: now 1770152879402298 us; error 31 us; skew 500 ppm
I20260203 21:07:59.404215 11154 webserver.cc:492] Webserver started at http://127.0.102.131:34935/ using document root <none> and password file <none>
I20260203 21:07:59.404870 11154 fs_manager.cc:362] Metadata directory not provided
I20260203 21:07:59.404932 11154 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:07:59.405133 11154 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:07:59.406919 11154 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "fee7db5ce97149a48ff7290469ceef66"
format_stamp: "Formatted at 2026-02-03 21:07:59 on dist-test-slave-j889"
server_key: "14b5256d8534909887a95df8a30748c1"
server_key_iv: "3eb9abb46cbdc41ba9275a797103bfb3"
server_key_version: "encryptionkey@0"
I20260203 21:07:59.407419 11154 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "fee7db5ce97149a48ff7290469ceef66"
format_stamp: "Formatted at 2026-02-03 21:07:59 on dist-test-slave-j889"
server_key: "14b5256d8534909887a95df8a30748c1"
server_key_iv: "3eb9abb46cbdc41ba9275a797103bfb3"
server_key_version: "encryptionkey@0"
I20260203 21:07:59.411092 11154 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260203 21:07:59.413499 11168 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:59.414603 11154 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:07:59.414703 11154 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "fee7db5ce97149a48ff7290469ceef66"
format_stamp: "Formatted at 2026-02-03 21:07:59 on dist-test-slave-j889"
server_key: "14b5256d8534909887a95df8a30748c1"
server_key_iv: "3eb9abb46cbdc41ba9275a797103bfb3"
server_key_version: "encryptionkey@0"
I20260203 21:07:59.414798 11154 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:07:59.419788 11154 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:07:59.420394 11154 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:07:59.420568 11154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:07:59.421152 11154 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:07:59.422137 11154 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:07:59.422206 11154 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:59.422246 11154 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:07:59.422274 11154 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:07:59.431360 11154 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:43305
I20260203 21:07:59.431396 11281 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:43305 every 8 connection(s)
I20260203 21:07:59.432345 11154 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
I20260203 21:07:59.437769 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:59.434072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:38923 (local address 127.0.102.190:45105)
0203 21:07:59.434243 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:59.434248 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:59.435222 (+ 974us) server_negotiation.cc:408] Connection header received
0203 21:07:59.435976 (+ 754us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:59.435979 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:59.436018 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:59.436134 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:59.437246 (+ 1112us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:07:59.437250 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:07:59.437252 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0203 21:07:59.437264 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:07:59.437290 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0203 21:07:59.437298 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:07:59.437389 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0203 21:07:59.437610 (+ 221us) server_negotiation.cc:300] Negotiation successful
0203 21:07:59.437657 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260203 21:07:59.437824 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 11154
I20260203 21:07:59.437945 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:07:59.438210 410 external_mini_cluster.cc:1468] Setting key 3e9f0f47af1ebab2ad8377d2892d62eb
I20260203 21:07:59.438660 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:07:59.434391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:45105 (local address 127.0.102.131:38923)
0203 21:07:59.435084 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0203 21:07:59.435121 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0203 21:07:59.435807 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:07:59.436324 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:07:59.436332 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:07:59.436421 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:07:59.436969 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0203 21:07:59.436980 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:07:59.437015 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0203 21:07:59.437074 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:07:59.437407 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:07:59.437415 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:07:59.437419 (+ 4us) client_negotiation.cc:769] Sending connection context
0203 21:07:59.437619 (+ 200us) client_negotiation.cc:240] Negotiation successful
0203 21:07:59.437861 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":96,"threads_started":1}
I20260203 21:07:59.439769 11282 heartbeater.cc:344] Connected to a master server at 127.0.102.190:45105
I20260203 21:07:59.439994 11282 heartbeater.cc:461] Registering TS with master...
I20260203 21:07:59.440443 11282 heartbeater.cc:507] Master 127.0.102.190:45105 requested a full tablet report, sending...
I20260203 21:07:59.441387 10832 ts_manager.cc:194] Registered new tserver with Master: fee7db5ce97149a48ff7290469ceef66 (127.0.102.131:43305)
I20260203 21:07:59.452324 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260203 21:07:59.455034 10871 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:07:59.454174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42824 (local address 127.0.102.190:45105)
0203 21:07:59.454303 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0203 21:07:59.454307 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:07:59.454450 (+ 143us) server_negotiation.cc:408] Connection header received
0203 21:07:59.454557 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:07:59.454559 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:07:59.454595 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:07:59.454660 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:07:59.454906 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42824: BlockingRecv error: recv got EOF from 127.0.0.1:42824 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260203 21:07:59.455250 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 10892
I20260203 21:07:59.460103 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 11023
I20260203 21:07:59.465574 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 11154
I20260203 21:07:59.470690 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 10801
2026-02-03T21:07:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (810 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:07:59 dist-test-slave-j889 krb5kdc[11292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:07:59 dist-test-slave-j889 krb5kdc[11292](info): set up 2 sockets
Feb 03 21:07:59 dist-test-slave-j889 krb5kdc[11292](info): commencing operation
krb5kdc: starting...
W20260203 21:08:01.321178 410 mini_kdc.cc:121] Time spent starting KDC: real 1.830s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:08:01 dist-test-slave-j889 krb5kdc[11292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:08:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:08:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:08:01.460992 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:41083
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:40313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:41083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:08:01.570528 11308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:08:01.570796 11308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:08:01.570873 11308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:08:01.574240 11308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:08:01.574306 11308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:08:01.574337 11308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:08:01.574369 11308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:08:01.574401 11308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:08:01.578665 11308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:40313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:41083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:41083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:08:01.579702 11308 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:08:01.580533 11308 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:08:01.586072 11314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:08:01.586118 11316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:08:01.586263 11308 server_base.cc:1060] running on GCE node
W20260203 21:08:01.586129 11313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:08:01.586813 11308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:08:01.587749 11308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:08:01.588886 11308 hybrid_clock.cc:648] HybridClock initialized: now 1770152881588868 us; error 35 us; skew 500 ppm
Feb 03 21:08:01 dist-test-slave-j889 krb5kdc[11292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:08:01.591464 11308 init.cc:377] Logged in from keytab as kudu/127.0.102.190@KRBTEST.COM (short username kudu)
I20260203 21:08:01.592526 11308 webserver.cc:492] Webserver started at http://127.0.102.190:46579/ using document root <none> and password file <none>
I20260203 21:08:01.593072 11308 fs_manager.cc:362] Metadata directory not provided
I20260203 21:08:01.593122 11308 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:08:01.593297 11308 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:08:01.594970 11308 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "b561cca143a94976b8dfa9c6442e02b8"
format_stamp: "Formatted at 2026-02-03 21:08:01 on dist-test-slave-j889"
server_key: "bd8c251f196d113c9405388984cfb126"
server_key_iv: "a04c8495c2829e72a63326407e1ca083"
server_key_version: "encryptionkey@0"
I20260203 21:08:01.595428 11308 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "b561cca143a94976b8dfa9c6442e02b8"
format_stamp: "Formatted at 2026-02-03 21:08:01 on dist-test-slave-j889"
server_key: "bd8c251f196d113c9405388984cfb126"
server_key_iv: "a04c8495c2829e72a63326407e1ca083"
server_key_version: "encryptionkey@0"
I20260203 21:08:01.598758 11308 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260203 21:08:01.600981 11323 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:01.601950 11308 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:08:01.602061 11308 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "b561cca143a94976b8dfa9c6442e02b8"
format_stamp: "Formatted at 2026-02-03 21:08:01 on dist-test-slave-j889"
server_key: "bd8c251f196d113c9405388984cfb126"
server_key_iv: "a04c8495c2829e72a63326407e1ca083"
server_key_version: "encryptionkey@0"
I20260203 21:08:01.602149 11308 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:08:01.620868 11308 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:08:01.628103 11308 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:08:01.628309 11308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:08:01.635382 11308 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:41083
I20260203 21:08:01.635376 11375 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:41083 every 8 connection(s)
I20260203 21:08:01.636438 11308 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:08:01.636840 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 11308
I20260203 21:08:01.636968 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:08:01.637202 410 external_mini_cluster.cc:1468] Setting key 97a60f3533473b16be2f12a3aee59b0c
I20260203 21:08:01.639720 11376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:08:01.645503 11376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8: Bootstrap starting.
Feb 03 21:08:01 dist-test-slave-j889 krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:08:01.648015 11376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8: Neither blocks nor log segments found. Creating new log.
I20260203 21:08:01.648850 11376 log.cc:826] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8: Log is configured to *not* fsync() on all Append() calls
I20260203 21:08:01.650856 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:01.638596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.0.102.190:41083)
0203 21:08:01.639031 (+ 435us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:01.639042 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:01.639085 (+ 43us) server_negotiation.cc:408] Connection header received
0203 21:08:01.640418 (+ 1333us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:01.640444 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:01.640837 (+ 393us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:01.641200 (+ 363us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:08:01.642027 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:01.643221 (+ 1194us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:01.643877 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:01.644157 (+ 280us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:01.646064 (+ 1907us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:08:01.646105 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:08:01.646118 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:08:01.646160 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:08:01.648116 (+ 1956us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:01.648523 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:01.648528 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:01.648533 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:01.648659 (+ 126us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:01.648858 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:01.648861 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:01.648863 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:01.649295 (+ 432us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:08:01.649454 (+ 159us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:01.649820 (+ 366us) server_negotiation.cc:300] Negotiation successful
0203 21:08:01.650105 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":146,"threads_started":1}
I20260203 21:08:01.650930 11376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8: No bootstrap required, opened a new log
I20260203 21:08:01.653614 11376 raft_consensus.cc:359] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b561cca143a94976b8dfa9c6442e02b8" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 41083 } }
I20260203 21:08:01.653795 11376 raft_consensus.cc:385] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:08:01.653842 11376 raft_consensus.cc:740] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b561cca143a94976b8dfa9c6442e02b8, State: Initialized, Role: FOLLOWER
I20260203 21:08:01.654182 11376 consensus_queue.cc:260] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b561cca143a94976b8dfa9c6442e02b8" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 41083 } }
I20260203 21:08:01.654284 11376 raft_consensus.cc:399] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:08:01.654326 11376 raft_consensus.cc:493] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:08:01.654395 11376 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:08:01.655153 11376 raft_consensus.cc:515] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b561cca143a94976b8dfa9c6442e02b8" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 41083 } }
I20260203 21:08:01.655457 11376 leader_election.cc:304] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b561cca143a94976b8dfa9c6442e02b8; no voters:
I20260203 21:08:01.655704 11376 leader_election.cc:290] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:08:01.656059 11381 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:08:01.656237 11381 raft_consensus.cc:697] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [term 1 LEADER]: Becoming Leader. State: Replica: b561cca143a94976b8dfa9c6442e02b8, State: Running, Role: LEADER
I20260203 21:08:01.656507 11381 consensus_queue.cc:237] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b561cca143a94976b8dfa9c6442e02b8" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 41083 } }
I20260203 21:08:01.656868 11376 sys_catalog.cc:565] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:08:01.657989 11382 sys_catalog.cc:455] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b561cca143a94976b8dfa9c6442e02b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b561cca143a94976b8dfa9c6442e02b8" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 41083 } } }
I20260203 21:08:01.658056 11383 sys_catalog.cc:455] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b561cca143a94976b8dfa9c6442e02b8. Latest consensus state: current_term: 1 leader_uuid: "b561cca143a94976b8dfa9c6442e02b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b561cca143a94976b8dfa9c6442e02b8" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 41083 } } }
I20260203 21:08:01.658100 11382 sys_catalog.cc:458] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [sys.catalog]: This master's current role is: LEADER
I20260203 21:08:01.658155 11383 sys_catalog.cc:458] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8 [sys.catalog]: This master's current role is: LEADER
I20260203 21:08:01.658458 11390 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:08:01.661497 11390 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:08:01.667513 11390 catalog_manager.cc:1357] Generated new cluster ID: 6f74932aa30c4a59ba0bb8783d331a82
I20260203 21:08:01.667618 11390 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:08:01.684206 11390 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:08:01.685218 11390 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:08:01.692126 11390 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8: Generated new TSK 0
I20260203 21:08:01.693017 11390 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:08:01.752349 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:41083
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:40313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:08:01.860523 11404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:08:01.860792 11404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:08:01.860853 11404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:08:01.864183 11404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:08:01.864265 11404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:08:01.864348 11404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:08:01.868829 11404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:40313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:41083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:08:01.869902 11404 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:08:01.870741 11404 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:08:01.877308 11409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:08:01.877314 11410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:08:01.877471 11412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:08:01.877549 11404 server_base.cc:1060] running on GCE node
I20260203 21:08:01.877908 11404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:08:01.878433 11404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:08:01.879598 11404 hybrid_clock.cc:648] HybridClock initialized: now 1770152881879577 us; error 35 us; skew 500 ppm
Feb 03 21:08:01 dist-test-slave-j889 krb5kdc[11292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:08:01.882154 11404 init.cc:377] Logged in from keytab as kudu/127.0.102.129@KRBTEST.COM (short username kudu)
I20260203 21:08:01.883239 11404 webserver.cc:492] Webserver started at http://127.0.102.129:40501/ using document root <none> and password file <none>
I20260203 21:08:01.883772 11404 fs_manager.cc:362] Metadata directory not provided
I20260203 21:08:01.883816 11404 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:08:01.883968 11404 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:08:01.885655 11404 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "facfff52ce0a4478b5077ed331fa6642"
format_stamp: "Formatted at 2026-02-03 21:08:01 on dist-test-slave-j889"
server_key: "e477ca34b45b1d07ecc08c05151d129c"
server_key_iv: "1513b3dc7c4af9047487dae8d7bc2d7b"
server_key_version: "encryptionkey@0"
I20260203 21:08:01.886070 11404 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "facfff52ce0a4478b5077ed331fa6642"
format_stamp: "Formatted at 2026-02-03 21:08:01 on dist-test-slave-j889"
server_key: "e477ca34b45b1d07ecc08c05151d129c"
server_key_iv: "1513b3dc7c4af9047487dae8d7bc2d7b"
server_key_version: "encryptionkey@0"
I20260203 21:08:01.889516 11404 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260203 21:08:01.891665 11419 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:01.892755 11404 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:08:01.892851 11404 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "facfff52ce0a4478b5077ed331fa6642"
format_stamp: "Formatted at 2026-02-03 21:08:01 on dist-test-slave-j889"
server_key: "e477ca34b45b1d07ecc08c05151d129c"
server_key_iv: "1513b3dc7c4af9047487dae8d7bc2d7b"
server_key_version: "encryptionkey@0"
I20260203 21:08:01.892948 11404 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:08:01.912566 11404 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:08:01.915599 11404 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:08:01.915783 11404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:08:01.916329 11404 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:08:01.917250 11404 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:08:01.917297 11404 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:01.917351 11404 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:08:01.917378 11404 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:01.926666 11404 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:33577
I20260203 21:08:01.926690 11532 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:33577 every 8 connection(s)
I20260203 21:08:01.927672 11404 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:08:01.928000 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 11404
I20260203 21:08:01.928139 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:08:01.928421 410 external_mini_cluster.cc:1468] Setting key ce5de01e9e71372dc6eaa62f3f3738b6
Feb 03 21:08:01 dist-test-slave-j889 krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.130@KRBTEST.COM" created.
I20260203 21:08:01.939854 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:01.929467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:53723 (local address 127.0.102.190:41083)
0203 21:08:01.929596 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:01.929601 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:01.930427 (+ 826us) server_negotiation.cc:408] Connection header received
0203 21:08:01.931241 (+ 814us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:01.931244 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:01.931297 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:01.931369 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:08:01.933078 (+ 1709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:01.933586 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:01.934202 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:01.934380 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:01.936513 (+ 2133us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:08:01.936534 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:08:01.936536 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:08:01.936563 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:08:01.937871 (+ 1308us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:01.938333 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:01.938337 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:01.938339 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:01.938386 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:01.938646 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:01.938650 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:01.938652 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:01.938842 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:08:01.938930 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:01.939543 (+ 613us) server_negotiation.cc:300] Negotiation successful
0203 21:08:01.939660 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260203 21:08:01.940505 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:01.929803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:41083 (local address 127.0.102.129:53723)
0203 21:08:01.930285 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:01.930320 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0203 21:08:01.931053 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:08:01.931531 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:08:01.931540 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:08:01.931940 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:08:01.932929 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:01.932945 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:01.933708 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:08:01.933711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:08:01.934097 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:01.934104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:01.934313 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:01.934969 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:08:01.934994 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:08:01.936364 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:08:01.937992 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:08:01.937997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:08:01.938010 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:08:01.938242 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:08:01.938472 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:08:01.938476 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:08:01.938477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:08:01.938566 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:08:01.938943 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:08:01.938951 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:08:01.939240 (+ 289us) client_negotiation.cc:769] Sending connection context
0203 21:08:01.939481 (+ 241us) client_negotiation.cc:240] Negotiation successful
0203 21:08:01.939768 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":125,"threads_started":1}
I20260203 21:08:01.941730 11533 heartbeater.cc:344] Connected to a master server at 127.0.102.190:41083
I20260203 21:08:01.941977 11533 heartbeater.cc:461] Registering TS with master...
I20260203 21:08:01.942533 11533 heartbeater.cc:507] Master 127.0.102.190:41083 requested a full tablet report, sending...
I20260203 21:08:01.944059 11340 ts_manager.cc:194] Registered new tserver with Master: facfff52ce0a4478b5077ed331fa6642 (127.0.102.129:33577)
I20260203 21:08:01.945299 11340 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:53723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:08:01.976254 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:41083
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:40313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:08:02.082060 11540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:08:02.082293 11540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:08:02.082355 11540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:08:02.086021 11540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:08:02.086100 11540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:08:02.086182 11540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:08:02.090664 11540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:40313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:41083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:08:02.091773 11540 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:08:02.092603 11540 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:08:02.099221 11548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:08:02.099222 11545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:08:02.099232 11546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:08:02.099550 11540 server_base.cc:1060] running on GCE node
I20260203 21:08:02.099946 11540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:08:02.100477 11540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:08:02.101651 11540 hybrid_clock.cc:648] HybridClock initialized: now 1770152882101632 us; error 35 us; skew 500 ppm
Feb 03 21:08:02 dist-test-slave-j889 krb5kdc[11292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:08:02.104167 11540 init.cc:377] Logged in from keytab as kudu/127.0.102.130@KRBTEST.COM (short username kudu)
I20260203 21:08:02.105274 11540 webserver.cc:492] Webserver started at http://127.0.102.130:45753/ using document root <none> and password file <none>
I20260203 21:08:02.105829 11540 fs_manager.cc:362] Metadata directory not provided
I20260203 21:08:02.105878 11540 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:08:02.106041 11540 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:08:02.107909 11540 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "c1602c16d97d422b9ecb8169424986b0"
format_stamp: "Formatted at 2026-02-03 21:08:02 on dist-test-slave-j889"
server_key: "d5a9c280b9bc8b3a200ebeb62d93c304"
server_key_iv: "c2746388994540cffc619c708f3b33a9"
server_key_version: "encryptionkey@0"
I20260203 21:08:02.108389 11540 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "c1602c16d97d422b9ecb8169424986b0"
format_stamp: "Formatted at 2026-02-03 21:08:02 on dist-test-slave-j889"
server_key: "d5a9c280b9bc8b3a200ebeb62d93c304"
server_key_iv: "c2746388994540cffc619c708f3b33a9"
server_key_version: "encryptionkey@0"
I20260203 21:08:02.111972 11540 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260203 21:08:02.114584 11555 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:02.115577 11540 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260203 21:08:02.115687 11540 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "c1602c16d97d422b9ecb8169424986b0"
format_stamp: "Formatted at 2026-02-03 21:08:02 on dist-test-slave-j889"
server_key: "d5a9c280b9bc8b3a200ebeb62d93c304"
server_key_iv: "c2746388994540cffc619c708f3b33a9"
server_key_version: "encryptionkey@0"
I20260203 21:08:02.115784 11540 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:08:02.135667 11540 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:08:02.138612 11540 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:08:02.138803 11540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:08:02.139382 11540 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:08:02.140297 11540 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:08:02.140344 11540 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:02.140391 11540 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:08:02.140420 11540 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:02.149587 11540 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:35863
I20260203 21:08:02.149612 11668 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:35863 every 8 connection(s)
I20260203 21:08:02.150883 11540 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:08:02.151788 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 11540
I20260203 21:08:02.151916 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:08:02.152170 410 external_mini_cluster.cc:1468] Setting key ff83e8aa9396a1100a24949c07b9e92e
Feb 03 21:08:02 dist-test-slave-j889 krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.131@KRBTEST.COM" created.
I20260203 21:08:02.164394 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.152786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:50581 (local address 127.0.102.190:41083)
0203 21:08:02.152944 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:02.152948 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:02.153671 (+ 723us) server_negotiation.cc:408] Connection header received
0203 21:08:02.154674 (+ 1003us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:02.154683 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:02.154730 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:02.154798 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:08:02.156520 (+ 1722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.157336 (+ 816us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:02.158094 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.158274 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:02.160850 (+ 2576us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:08:02.160870 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:08:02.160872 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:08:02.160898 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:08:02.162321 (+ 1423us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:02.162859 (+ 538us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:02.162863 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:02.162864 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:02.162912 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:02.163221 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:02.163224 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:02.163225 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:02.163425 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:08:02.163522 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:02.164140 (+ 618us) server_negotiation.cc:300] Negotiation successful
0203 21:08:02.164251 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260203 21:08:02.165355 11671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.153070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:41083 (local address 127.0.102.130:50581)
0203 21:08:02.153522 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:02.153569 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0203 21:08:02.154468 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:08:02.154965 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:08:02.154976 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:08:02.155448 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:08:02.156358 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:02.156371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.157566 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:08:02.157569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:08:02.157974 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:02.157982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.158220 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:02.158971 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:08:02.158998 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:08:02.160700 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:08:02.162450 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:08:02.162455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:08:02.162464 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:08:02.162760 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:08:02.162987 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:08:02.162990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:08:02.162991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:08:02.163091 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:08:02.163534 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:08:02.163541 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:08:02.163845 (+ 304us) client_negotiation.cc:769] Sending connection context
0203 21:08:02.164109 (+ 264us) client_negotiation.cc:240] Negotiation successful
0203 21:08:02.164442 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":95,"threads_started":1}
I20260203 21:08:02.166694 11669 heartbeater.cc:344] Connected to a master server at 127.0.102.190:41083
I20260203 21:08:02.166965 11669 heartbeater.cc:461] Registering TS with master...
I20260203 21:08:02.167546 11669 heartbeater.cc:507] Master 127.0.102.190:41083 requested a full tablet report, sending...
I20260203 21:08:02.168787 11340 ts_manager.cc:194] Registered new tserver with Master: c1602c16d97d422b9ecb8169424986b0 (127.0.102.130:35863)
I20260203 21:08:02.169531 11340 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.130@KRBTEST.COM'} at 127.0.102.130:50581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:08:02.202008 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:41083
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:40313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:08:02.309454 11676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:08:02.309715 11676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:08:02.309778 11676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:08:02.313299 11676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:08:02.313374 11676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:08:02.313472 11676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:08:02.317966 11676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:40313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:41083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:08:02.319056 11676 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:08:02.319944 11676 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:08:02.326634 11682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:08:02.326625 11684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:08:02.326623 11681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:08:02.327054 11676 server_base.cc:1060] running on GCE node
I20260203 21:08:02.327407 11676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:08:02.327942 11676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:08:02.329080 11676 hybrid_clock.cc:648] HybridClock initialized: now 1770152882329062 us; error 33 us; skew 500 ppm
Feb 03 21:08:02 dist-test-slave-j889 krb5kdc[11292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:08:02.331561 11676 init.cc:377] Logged in from keytab as kudu/127.0.102.131@KRBTEST.COM (short username kudu)
I20260203 21:08:02.332584 11676 webserver.cc:492] Webserver started at http://127.0.102.131:41473/ using document root <none> and password file <none>
I20260203 21:08:02.333125 11676 fs_manager.cc:362] Metadata directory not provided
I20260203 21:08:02.333175 11676 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:08:02.333356 11676 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260203 21:08:02.335070 11676 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "e039e44547d8425fb631b87b477cbe2f"
format_stamp: "Formatted at 2026-02-03 21:08:02 on dist-test-slave-j889"
server_key: "3be608157f2e6c79910b704a2d4532de"
server_key_iv: "9d9f403b19f1aaa2feaa098ea084f189"
server_key_version: "encryptionkey@0"
I20260203 21:08:02.335541 11676 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "e039e44547d8425fb631b87b477cbe2f"
format_stamp: "Formatted at 2026-02-03 21:08:02 on dist-test-slave-j889"
server_key: "3be608157f2e6c79910b704a2d4532de"
server_key_iv: "9d9f403b19f1aaa2feaa098ea084f189"
server_key_version: "encryptionkey@0"
I20260203 21:08:02.338881 11676 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260203 21:08:02.341090 11691 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:02.342092 11676 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260203 21:08:02.342198 11676 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "e039e44547d8425fb631b87b477cbe2f"
format_stamp: "Formatted at 2026-02-03 21:08:02 on dist-test-slave-j889"
server_key: "3be608157f2e6c79910b704a2d4532de"
server_key_iv: "9d9f403b19f1aaa2feaa098ea084f189"
server_key_version: "encryptionkey@0"
I20260203 21:08:02.342286 11676 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:08:02.359539 11676 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:08:02.362198 11676 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:08:02.362375 11676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:08:02.362934 11676 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:08:02.363867 11676 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:08:02.363916 11676 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:02.363968 11676 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:08:02.363994 11676 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:02.372694 11676 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:37033
I20260203 21:08:02.372730 11804 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:37033 every 8 connection(s)
I20260203 21:08:02.373661 11676 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
I20260203 21:08:02.377867 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 11676
I20260203 21:08:02.378077 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:08:02.378353 410 external_mini_cluster.cc:1468] Setting key 11cc223f55044653bb215a60076f18f4
Feb 03 21:08:02 dist-test-slave-j889 krb5kdc[11292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:08:02.386504 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.375495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:42607 (local address 127.0.102.190:41083)
0203 21:08:02.375624 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:02.375628 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:02.376317 (+ 689us) server_negotiation.cc:408] Connection header received
0203 21:08:02.377178 (+ 861us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:02.377182 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:02.377245 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:02.377324 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:08:02.379229 (+ 1905us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.379894 (+ 665us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:02.380716 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.380905 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:02.383109 (+ 2204us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:08:02.383127 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:08:02.383129 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:08:02.383159 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:08:02.384505 (+ 1346us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:02.385030 (+ 525us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:02.385033 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:02.385035 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:02.385084 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:02.385359 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:02.385362 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:02.385363 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:02.385526 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:08:02.385612 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:02.386199 (+ 587us) server_negotiation.cc:300] Negotiation successful
0203 21:08:02.386334 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260203 21:08:02.387198 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.375767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:41083 (local address 127.0.102.131:42607)
0203 21:08:02.376186 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:02.376222 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0203 21:08:02.376984 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:08:02.377543 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:08:02.377552 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:08:02.377974 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:08:02.379060 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:02.379078 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.380016 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:08:02.380020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:08:02.380575 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:02.380585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.380846 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:02.381407 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:08:02.381435 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:08:02.382949 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:08:02.384649 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:08:02.384656 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:08:02.384671 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:08:02.384930 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:08:02.385166 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:08:02.385169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:08:02.385171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:08:02.385269 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:08:02.385623 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:08:02.385632 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:08:02.385920 (+ 288us) client_negotiation.cc:769] Sending connection context
0203 21:08:02.386142 (+ 222us) client_negotiation.cc:240] Negotiation successful
0203 21:08:02.386433 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260203 21:08:02.388556 11805 heartbeater.cc:344] Connected to a master server at 127.0.102.190:41083
I20260203 21:08:02.388854 11805 heartbeater.cc:461] Registering TS with master...
I20260203 21:08:02.389417 11805 heartbeater.cc:507] Master 127.0.102.190:41083 requested a full tablet report, sending...
I20260203 21:08:02.390467 11340 ts_manager.cc:194] Registered new tserver with Master: e039e44547d8425fb631b87b477cbe2f (127.0.102.131:37033)
I20260203 21:08:02.391134 11340 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.131@KRBTEST.COM'} at 127.0.102.131:42607
I20260203 21:08:02.392328 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260203 21:08:02.400437 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.394108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55188 (local address 127.0.102.190:41083)
0203 21:08:02.394222 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:02.394225 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:02.394351 (+ 126us) server_negotiation.cc:408] Connection header received
0203 21:08:02.394471 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:02.394474 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:02.394514 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:02.394580 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:08:02.395386 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.395925 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:02.396516 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.396708 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:02.397744 (+ 1036us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:08:02.397762 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:08:02.397764 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:08:02.397789 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:08:02.399131 (+ 1342us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:02.399508 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:02.399511 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:02.399512 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:02.399574 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:02.399752 (+ 178us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:02.399754 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:02.399756 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:02.399933 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:08:02.400032 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:02.400202 (+ 170us) server_negotiation.cc:300] Negotiation successful
0203 21:08:02.400305 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260203 21:08:02.402943 11340 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260203 21:08:02.405357 11340 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:08:02.418198 11816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.413675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:37033 (local address 127.0.0.1:52592)
0203 21:08:02.414028 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:02.414085 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0203 21:08:02.414240 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:08:02.415554 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:08:02.415560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:08:02.415580 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:08:02.415836 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:02.415842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.416960 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:08:02.416963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:08:02.417669 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:02.417679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.417798 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:02.417857 (+ 59us) client_negotiation.cc:769] Sending connection context
0203 21:08:02.417961 (+ 104us) client_negotiation.cc:240] Negotiation successful
0203 21:08:02.418057 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":104,"threads_started":1}
I20260203 21:08:02.419076 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.413939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:33577 (local address 127.0.0.1:42120)
0203 21:08:02.414229 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:02.414244 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0203 21:08:02.414338 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:08:02.416163 (+ 1825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:08:02.416166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:08:02.416186 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:08:02.416408 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:02.416416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.417872 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:08:02.417882 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:08:02.418772 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:02.418781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.418856 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:02.418872 (+ 16us) client_negotiation.cc:769] Sending connection context
0203 21:08:02.418908 (+ 36us) client_negotiation.cc:240] Negotiation successful
0203 21:08:02.418959 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":107,"threads_started":1}
I20260203 21:08:02.419358 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.413845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52592 (local address 127.0.102.131:37033)
0203 21:08:02.414884 (+ 1039us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:02.414897 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:02.414923 (+ 26us) server_negotiation.cc:408] Connection header received
0203 21:08:02.415019 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:02.415025 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:02.415223 (+ 198us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:02.415365 (+ 142us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:08:02.415952 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.416857 (+ 905us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:02.418558 (+ 1701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.418949 (+ 391us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:02.419072 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:02.419136 (+ 64us) server_negotiation.cc:300] Negotiation successful
0203 21:08:02.419223 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":130,"threads_started":1}
I20260203 21:08:02.419198 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.414847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:35863 (local address 127.0.0.1:51360)
0203 21:08:02.415489 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:02.415502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0203 21:08:02.415573 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:08:02.416336 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:08:02.416339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:08:02.416357 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:08:02.416737 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:02.416747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.418096 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:08:02.418099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:08:02.418908 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:02.418916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.419004 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:02.419017 (+ 13us) client_negotiation.cc:769] Sending connection context
0203 21:08:02.419058 (+ 41us) client_negotiation.cc:240] Negotiation successful
0203 21:08:02.419103 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":111,"threads_started":1}
I20260203 21:08:02.420006 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.415261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.0.102.130:35863)
0203 21:08:02.415728 (+ 467us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:02.415736 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:02.415761 (+ 25us) server_negotiation.cc:408] Connection header received
0203 21:08:02.415848 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:02.415854 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:02.416024 (+ 170us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:02.416170 (+ 146us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:08:02.416875 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.417991 (+ 1116us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:02.419043 (+ 1052us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.419549 (+ 506us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:02.419650 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:02.419769 (+ 119us) server_negotiation.cc:300] Negotiation successful
0203 21:08:02.419866 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":110,"threads_started":1}
I20260203 21:08:02.420480 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.414495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42120 (local address 127.0.102.129:33577)
0203 21:08:02.415736 (+ 1241us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:02.415742 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:02.415761 (+ 19us) server_negotiation.cc:408] Connection header received
0203 21:08:02.415848 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:02.415854 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:02.416024 (+ 170us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:02.416170 (+ 146us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:08:02.416555 (+ 385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.417749 (+ 1194us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:02.419212 (+ 1463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.420161 (+ 949us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:02.420258 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:02.420309 (+ 51us) server_negotiation.cc:300] Negotiation successful
0203 21:08:02.420370 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1149,"thread_start_us":124,"threads_started":1}
I20260203 21:08:02.422502 11467 tablet_service.cc:1505] Processing CreateTablet for tablet d2bfb988fa5641ee9f04becfe89ce075 (DEFAULT_TABLE table=test-table [id=1b210e68f9734237923ac7bd488aef1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:08:02.422464 11739 tablet_service.cc:1505] Processing CreateTablet for tablet d2bfb988fa5641ee9f04becfe89ce075 (DEFAULT_TABLE table=test-table [id=1b210e68f9734237923ac7bd488aef1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:08:02.422461 11603 tablet_service.cc:1505] Processing CreateTablet for tablet d2bfb988fa5641ee9f04becfe89ce075 (DEFAULT_TABLE table=test-table [id=1b210e68f9734237923ac7bd488aef1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:08:02.423449 11467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2bfb988fa5641ee9f04becfe89ce075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:08:02.423451 11739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2bfb988fa5641ee9f04becfe89ce075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:08:02.423465 11603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2bfb988fa5641ee9f04becfe89ce075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:08:02.429221 11822 tablet_bootstrap.cc:492] T d2bfb988fa5641ee9f04becfe89ce075 P facfff52ce0a4478b5077ed331fa6642: Bootstrap starting.
I20260203 21:08:02.430451 11823 tablet_bootstrap.cc:492] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f: Bootstrap starting.
I20260203 21:08:02.430451 11824 tablet_bootstrap.cc:492] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0: Bootstrap starting.
I20260203 21:08:02.431475 11822 tablet_bootstrap.cc:654] T d2bfb988fa5641ee9f04becfe89ce075 P facfff52ce0a4478b5077ed331fa6642: Neither blocks nor log segments found. Creating new log.
I20260203 21:08:02.432505 11824 tablet_bootstrap.cc:654] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0: Neither blocks nor log segments found. Creating new log.
I20260203 21:08:02.432736 11822 log.cc:826] T d2bfb988fa5641ee9f04becfe89ce075 P facfff52ce0a4478b5077ed331fa6642: Log is configured to *not* fsync() on all Append() calls
I20260203 21:08:02.433180 11823 tablet_bootstrap.cc:654] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f: Neither blocks nor log segments found. Creating new log.
I20260203 21:08:02.433321 11824 log.cc:826] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0: Log is configured to *not* fsync() on all Append() calls
I20260203 21:08:02.433945 11823 log.cc:826] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f: Log is configured to *not* fsync() on all Append() calls
I20260203 21:08:02.435263 11824 tablet_bootstrap.cc:492] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0: No bootstrap required, opened a new log
I20260203 21:08:02.435427 11822 tablet_bootstrap.cc:492] T d2bfb988fa5641ee9f04becfe89ce075 P facfff52ce0a4478b5077ed331fa6642: No bootstrap required, opened a new log
I20260203 21:08:02.435482 11824 ts_tablet_manager.cc:1403] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260203 21:08:02.435600 11822 ts_tablet_manager.cc:1403] T d2bfb988fa5641ee9f04becfe89ce075 P facfff52ce0a4478b5077ed331fa6642: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260203 21:08:02.435702 11823 tablet_bootstrap.cc:492] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f: No bootstrap required, opened a new log
I20260203 21:08:02.435897 11823 ts_tablet_manager.cc:1403] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260203 21:08:02.438944 11822 raft_consensus.cc:359] T d2bfb988fa5641ee9f04becfe89ce075 P facfff52ce0a4478b5077ed331fa6642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039e44547d8425fb631b87b477cbe2f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 37033 } } peers { permanent_uuid: "facfff52ce0a4478b5077ed331fa6642" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 33577 } } peers { permanent_uuid: "c1602c16d97d422b9ecb8169424986b0" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35863 } }
I20260203 21:08:02.439188 11822 raft_consensus.cc:385] T d2bfb988fa5641ee9f04becfe89ce075 P facfff52ce0a4478b5077ed331fa6642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:08:02.439244 11822 raft_consensus.cc:740] T d2bfb988fa5641ee9f04becfe89ce075 P facfff52ce0a4478b5077ed331fa6642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: facfff52ce0a4478b5077ed331fa6642, State: Initialized, Role: FOLLOWER
I20260203 21:08:02.439110 11824 raft_consensus.cc:359] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039e44547d8425fb631b87b477cbe2f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 37033 } } peers { permanent_uuid: "facfff52ce0a4478b5077ed331fa6642" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 33577 } } peers { permanent_uuid: "c1602c16d97d422b9ecb8169424986b0" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35863 } }
I20260203 21:08:02.439325 11824 raft_consensus.cc:385] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:08:02.439378 11824 raft_consensus.cc:740] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1602c16d97d422b9ecb8169424986b0, State: Initialized, Role: FOLLOWER
I20260203 21:08:02.439715 11822 consensus_queue.cc:260] T d2bfb988fa5641ee9f04becfe89ce075 P facfff52ce0a4478b5077ed331fa6642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039e44547d8425fb631b87b477cbe2f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 37033 } } peers { permanent_uuid: "facfff52ce0a4478b5077ed331fa6642" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 33577 } } peers { permanent_uuid: "c1602c16d97d422b9ecb8169424986b0" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35863 } }
I20260203 21:08:02.440183 11824 consensus_queue.cc:260] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039e44547d8425fb631b87b477cbe2f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 37033 } } peers { permanent_uuid: "facfff52ce0a4478b5077ed331fa6642" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 33577 } } peers { permanent_uuid: "c1602c16d97d422b9ecb8169424986b0" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35863 } }
I20260203 21:08:02.440297 11823 raft_consensus.cc:359] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039e44547d8425fb631b87b477cbe2f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 37033 } } peers { permanent_uuid: "facfff52ce0a4478b5077ed331fa6642" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 33577 } } peers { permanent_uuid: "c1602c16d97d422b9ecb8169424986b0" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35863 } }
I20260203 21:08:02.440495 11823 raft_consensus.cc:385] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:08:02.440557 11823 raft_consensus.cc:740] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e039e44547d8425fb631b87b477cbe2f, State: Initialized, Role: FOLLOWER
I20260203 21:08:02.440726 11822 ts_tablet_manager.cc:1434] T d2bfb988fa5641ee9f04becfe89ce075 P facfff52ce0a4478b5077ed331fa6642: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260203 21:08:02.441126 11533 heartbeater.cc:499] Master 127.0.102.190:41083 was elected leader, sending a full tablet report...
I20260203 21:08:02.441238 11824 ts_tablet_manager.cc:1434] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260203 21:08:02.441563 11669 heartbeater.cc:499] Master 127.0.102.190:41083 was elected leader, sending a full tablet report...
I20260203 21:08:02.442031 11823 consensus_queue.cc:260] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039e44547d8425fb631b87b477cbe2f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 37033 } } peers { permanent_uuid: "facfff52ce0a4478b5077ed331fa6642" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 33577 } } peers { permanent_uuid: "c1602c16d97d422b9ecb8169424986b0" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35863 } }
I20260203 21:08:02.442678 11805 heartbeater.cc:499] Master 127.0.102.190:41083 was elected leader, sending a full tablet report...
I20260203 21:08:02.442829 11823 ts_tablet_manager.cc:1434] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260203 21:08:02.448846 11830 raft_consensus.cc:493] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:08:02.448985 11830 raft_consensus.cc:515] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039e44547d8425fb631b87b477cbe2f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 37033 } } peers { permanent_uuid: "facfff52ce0a4478b5077ed331fa6642" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 33577 } } peers { permanent_uuid: "c1602c16d97d422b9ecb8169424986b0" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35863 } }
I20260203 21:08:02.450022 11830 leader_election.cc:290] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers facfff52ce0a4478b5077ed331fa6642 (127.0.102.129:33577), c1602c16d97d422b9ecb8169424986b0 (127.0.102.130:35863)
I20260203 21:08:02.453017 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.450184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:33577 (local address 127.0.102.131:44211)
0203 21:08:02.450355 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:02.450371 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0203 21:08:02.450459 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:08:02.450730 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:08:02.450733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:08:02.450759 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:08:02.451024 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:02.451032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.451878 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:08:02.451882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:08:02.452613 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:02.452623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.452749 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:02.452771 (+ 22us) client_negotiation.cc:769] Sending connection context
0203 21:08:02.452830 (+ 59us) client_negotiation.cc:240] Negotiation successful
0203 21:08:02.452892 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260203 21:08:02.453540 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.450324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:44211 (local address 127.0.102.129:33577)
0203 21:08:02.450490 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:02.450495 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:02.450511 (+ 16us) server_negotiation.cc:408] Connection header received
0203 21:08:02.450564 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:02.450567 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:02.450622 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:02.450698 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:08:02.451185 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.451781 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:02.452778 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.453283 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:02.453321 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:02.453369 (+ 48us) server_negotiation.cc:300] Negotiation successful
0203 21:08:02.453426 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260203 21:08:02.454170 11487 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2bfb988fa5641ee9f04becfe89ce075" candidate_uuid: "e039e44547d8425fb631b87b477cbe2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "facfff52ce0a4478b5077ed331fa6642" is_pre_election: true
I20260203 21:08:02.454245 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.450656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:58591 (local address 127.0.102.130:35863)
0203 21:08:02.450808 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:02.450812 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:02.450824 (+ 12us) server_negotiation.cc:408] Connection header received
0203 21:08:02.450868 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:02.450871 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:02.450913 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:02.450983 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:08:02.451430 (+ 447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.451965 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:02.453132 (+ 1167us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.453636 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:02.453671 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:02.454053 (+ 382us) server_negotiation.cc:300] Negotiation successful
0203 21:08:02.454135 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260203 21:08:02.454447 11487 raft_consensus.cc:2468] T d2bfb988fa5641ee9f04becfe89ce075 P facfff52ce0a4478b5077ed331fa6642 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e039e44547d8425fb631b87b477cbe2f in term 0.
I20260203 21:08:02.454581 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.450254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:35863 (local address 127.0.102.131:58591)
0203 21:08:02.450580 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:02.450592 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0203 21:08:02.450682 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:08:02.451011 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:08:02.451013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:08:02.451028 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:08:02.451322 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:02.451330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.452090 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:08:02.452094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:08:02.452993 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:02.453003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.453906 (+ 903us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:02.453924 (+ 18us) client_negotiation.cc:769] Sending connection context
0203 21:08:02.454364 (+ 440us) client_negotiation.cc:240] Negotiation successful
0203 21:08:02.454426 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":131,"threads_started":1}
I20260203 21:08:02.454926 11693 leader_election.cc:304] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e039e44547d8425fb631b87b477cbe2f, facfff52ce0a4478b5077ed331fa6642; no voters:
I20260203 21:08:02.455165 11830 raft_consensus.cc:2804] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:08:02.455155 11623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2bfb988fa5641ee9f04becfe89ce075" candidate_uuid: "e039e44547d8425fb631b87b477cbe2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1602c16d97d422b9ecb8169424986b0" is_pre_election: true
I20260203 21:08:02.455241 11830 raft_consensus.cc:493] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:08:02.455291 11830 raft_consensus.cc:3060] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:08:02.455390 11623 raft_consensus.cc:2468] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e039e44547d8425fb631b87b477cbe2f in term 0.
I20260203 21:08:02.456362 11830 raft_consensus.cc:515] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039e44547d8425fb631b87b477cbe2f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 37033 } } peers { permanent_uuid: "facfff52ce0a4478b5077ed331fa6642" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 33577 } } peers { permanent_uuid: "c1602c16d97d422b9ecb8169424986b0" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35863 } }
I20260203 21:08:02.456727 11830 leader_election.cc:290] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [CANDIDATE]: Term 1 election: Requested vote from peers facfff52ce0a4478b5077ed331fa6642 (127.0.102.129:33577), c1602c16d97d422b9ecb8169424986b0 (127.0.102.130:35863)
I20260203 21:08:02.457032 11487 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2bfb988fa5641ee9f04becfe89ce075" candidate_uuid: "e039e44547d8425fb631b87b477cbe2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "facfff52ce0a4478b5077ed331fa6642"
I20260203 21:08:02.457088 11623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2bfb988fa5641ee9f04becfe89ce075" candidate_uuid: "e039e44547d8425fb631b87b477cbe2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1602c16d97d422b9ecb8169424986b0"
I20260203 21:08:02.457145 11487 raft_consensus.cc:3060] T d2bfb988fa5641ee9f04becfe89ce075 P facfff52ce0a4478b5077ed331fa6642 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:08:02.457202 11623 raft_consensus.cc:3060] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:08:02.458125 11623 raft_consensus.cc:2468] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e039e44547d8425fb631b87b477cbe2f in term 1.
I20260203 21:08:02.458137 11487 raft_consensus.cc:2468] T d2bfb988fa5641ee9f04becfe89ce075 P facfff52ce0a4478b5077ed331fa6642 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e039e44547d8425fb631b87b477cbe2f in term 1.
I20260203 21:08:02.458388 11692 leader_election.cc:304] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c1602c16d97d422b9ecb8169424986b0, e039e44547d8425fb631b87b477cbe2f, facfff52ce0a4478b5077ed331fa6642; no voters:
I20260203 21:08:02.458603 11830 raft_consensus.cc:2804] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:08:02.458840 11830 raft_consensus.cc:697] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [term 1 LEADER]: Becoming Leader. State: Replica: e039e44547d8425fb631b87b477cbe2f, State: Running, Role: LEADER
I20260203 21:08:02.459160 11830 consensus_queue.cc:237] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039e44547d8425fb631b87b477cbe2f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 37033 } } peers { permanent_uuid: "facfff52ce0a4478b5077ed331fa6642" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 33577 } } peers { permanent_uuid: "c1602c16d97d422b9ecb8169424986b0" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35863 } }
I20260203 21:08:02.462244 11338 catalog_manager.cc:5654] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f reported cstate change: term changed from 0 to 1, leader changed from <none> to e039e44547d8425fb631b87b477cbe2f (127.0.102.131). New cstate: current_term: 1 leader_uuid: "e039e44547d8425fb631b87b477cbe2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e039e44547d8425fb631b87b477cbe2f" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 37033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "facfff52ce0a4478b5077ed331fa6642" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 33577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1602c16d97d422b9ecb8169424986b0" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35863 } health_report { overall_health: UNKNOWN } } }
I20260203 21:08:02.472769 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.469601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.0.102.131:37033)
0203 21:08:02.469733 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:02.469736 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:02.469750 (+ 14us) server_negotiation.cc:408] Connection header received
0203 21:08:02.469863 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:02.469865 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:02.469926 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:02.470005 (+ 79us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:08:02.470416 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.470955 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:02.471643 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:02.471799 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:02.471894 (+ 95us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:08:02.472361 (+ 467us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:08:02.472462 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:02.472556 (+ 94us) server_negotiation.cc:300] Negotiation successful
0203 21:08:02.472606 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260203 21:08:02.478480 11487 raft_consensus.cc:1275] T d2bfb988fa5641ee9f04becfe89ce075 P facfff52ce0a4478b5077ed331fa6642 [term 1 FOLLOWER]: Refusing update from remote peer e039e44547d8425fb631b87b477cbe2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:08:02.478480 11623 raft_consensus.cc:1275] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0 [term 1 FOLLOWER]: Refusing update from remote peer e039e44547d8425fb631b87b477cbe2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:08:02.479143 11832 consensus_queue.cc:1048] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "facfff52ce0a4478b5077ed331fa6642" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 33577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:08:02.479322 11830 consensus_queue.cc:1048] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1602c16d97d422b9ecb8169424986b0" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 35863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:08:02.489228 11836 mvcc.cc:204] Tried to move back new op lower bound from 7250546206624813056 to 7250546206555443200. Current Snapshot: MvccSnapshot[applied={T|T < 7250546206624813056 or (T in {7250546206624813056})}]
I20260203 21:08:02.499123 11340 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:08:02.499290 11340 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:08:02.501758 11340 catalog_manager.cc:5941] T 00000000000000000000000000000000 P b561cca143a94976b8dfa9c6442e02b8: Sending DeleteTablet for 3 replicas of tablet d2bfb988fa5641ee9f04becfe89ce075
I20260203 21:08:02.502688 11603 tablet_service.cc:1552] Processing DeleteTablet for tablet d2bfb988fa5641ee9f04becfe89ce075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:08:02 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:51360
I20260203 21:08:02.502688 11739 tablet_service.cc:1552] Processing DeleteTablet for tablet d2bfb988fa5641ee9f04becfe89ce075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:08:02 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:52592
I20260203 21:08:02.502696 11467 tablet_service.cc:1552] Processing DeleteTablet for tablet d2bfb988fa5641ee9f04becfe89ce075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:08:02 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:42120
I20260203 21:08:02.503338 11846 tablet_replica.cc:333] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0: stopping tablet replica
I20260203 21:08:02.503645 11846 raft_consensus.cc:2243] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:08:02.503846 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 11404
I20260203 21:08:02.504019 11846 raft_consensus.cc:2272] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:08:02.504703 11845 tablet_replica.cc:333] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f: stopping tablet replica
I20260203 21:08:02.504977 11845 raft_consensus.cc:2243] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [term 1 LEADER]: Raft consensus shutting down.
I20260203 21:08:02.505448 11845 raft_consensus.cc:2272] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:08:02.506870 11845 ts_tablet_manager.cc:1916] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:08:02.506870 11846 ts_tablet_manager.cc:1916] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:08:02.509961 11845 ts_tablet_manager.cc:1929] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:08:02.509953 11846 ts_tablet_manager.cc:1929] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:08:02.510074 11846 log.cc:1199] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-1/wal/wals/d2bfb988fa5641ee9f04becfe89ce075
I20260203 21:08:02.510226 11845 log.cc:1199] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1770152788143875-410-0/minicluster-data/ts-2/wal/wals/d2bfb988fa5641ee9f04becfe89ce075
I20260203 21:08:02.510450 11846 ts_tablet_manager.cc:1950] T d2bfb988fa5641ee9f04becfe89ce075 P c1602c16d97d422b9ecb8169424986b0: Deleting consensus metadata
I20260203 21:08:02.510643 11845 ts_tablet_manager.cc:1950] T d2bfb988fa5641ee9f04becfe89ce075 P e039e44547d8425fb631b87b477cbe2f: Deleting consensus metadata
I20260203 21:08:02.511922 11327 catalog_manager.cc:4985] TS e039e44547d8425fb631b87b477cbe2f (127.0.102.131:37033): tablet d2bfb988fa5641ee9f04becfe89ce075 (table test-table [id=1b210e68f9734237923ac7bd488aef1f]) successfully deleted
I20260203 21:08:02.511922 11324 catalog_manager.cc:4985] TS c1602c16d97d422b9ecb8169424986b0 (127.0.102.130:35863): tablet d2bfb988fa5641ee9f04becfe89ce075 (table test-table [id=1b210e68f9734237923ac7bd488aef1f]) successfully deleted
W20260203 21:08:02.514170 11325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.102.129:33577 (error 108)
I20260203 21:08:02.514205 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 11540
I20260203 21:08:02.515650 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:02.515376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:33577 (local address 127.0.0.1:42124)
0203 21:08:02.515509 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:02.515566 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.129:33577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260203 21:08:02.515889 11325 catalog_manager.cc:4712] TS facfff52ce0a4478b5077ed331fa6642 (127.0.102.129:33577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2bfb988fa5641ee9f04becfe89ce075: Network error: Client connection negotiation failed: client connection to 127.0.102.129:33577: connect: Connection refused (error 111)
I20260203 21:08:02.521306 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 11676
I20260203 21:08:02.527045 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 11308
2026-02-03T21:08:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3058 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:08:02 dist-test-slave-j889 krb5kdc[11850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:08:02 dist-test-slave-j889 krb5kdc[11850](info): set up 2 sockets
Feb 03 21:08:02 dist-test-slave-j889 krb5kdc[11850](info): commencing operation
krb5kdc: starting...
W20260203 21:08:04.401190 410 mini_kdc.cc:121] Time spent starting KDC: real 1.851s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:08:04 dist-test-slave-j889 krb5kdc[11850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:08:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:08:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.102.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.102.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.102.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/rangeradmin_127.0.102.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.102.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/rangeradmin_127.0.102.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.102.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.102.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.102.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/rangerlookup_127.0.102.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.102.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/rangerlookup_127.0.102.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.102.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/HTTP_127.0.102.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.102.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/HTTP_127.0.102.148@KRBTEST.COM.keytab.
I20260203 21:08:04.585224 410 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3aGl8N/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/postgres -l logfile start
2026-02-03 21:08:07.899 UTC [11881] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-02-03 21:08:07.899 UTC [11881] LOG: listening on IPv4 address "127.0.102.148", port 43925
2026-02-03 21:08:07.903 UTC [11881] LOG: listening on Unix socket "/tmp/.s.PGSQL.43925"
2026-02-03 21:08:07.913 UTC [11886] LOG: database system was shut down at 2026-02-03 21:08:05 UTC
2026-02-03 21:08:07.918 UTC [11881] LOG: database system is ready to accept connections
I20260203 21:08:09.697016 410 mini_postgres.cc:96] Postgres bound to 43925
2026-02-03 21:08:09.702 UTC [11892] FATAL: database "slave" does not exist
127.0.102.148:43925 - accepting connections
I20260203 21:08:09.703619 410 mini_ranger.cc:161] Starting Ranger...
I20260203 21:08:09.722622 410 mini_ranger.cc:84] Created miniranger Postgres user
I20260203 21:08:09.793150 410 mini_ranger.cc:87] Created ranger Postgres database
I20260203 21:08:09.793257 410 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-admin
2026-02-03 21:08:10,070 [I] DB FLAVOR :POSTGRES
2026-02-03 21:08:10,071 [I] --------- Verifying Ranger DB connection ---------
2026-02-03 21:08:10,071 [I] Checking connection..
2026-02-03 21:08:10,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-02-03 21:08:10,389 [I] Checking connection passed.
2026-02-03 21:08:10,390 [I] --------- Verifying version history table ---------
2026-02-03 21:08:10,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-03 21:08:10,724 [I] Table x_db_version_h does not exist in database ranger
2026-02-03 21:08:10,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-03 21:08:11,040 [I] Table x_db_version_h does not exist in database ranger
2026-02-03 21:08:11,040 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-02-03 21:08:11,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-02-03 21:08:11.338 UTC [11959] WARNING: there is no transaction in progress
2026-02-03 21:08:11,344 [I] create_dbversion_catalog.sql file imported successfully
2026-02-03 21:08:11,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-03 21:08:11,664 [I] Table x_db_version_h already exists in database 'ranger'
2026-02-03 21:08:11,665 [I] --------- Importing Ranger Core DB Schema ---------
2026-02-03 21:08:11,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-02-03 21:08:11,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-02-03 21:08:12,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j889.c.gcp-upstream.internal','N') ;"
2026-02-03 21:08:12,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-02-03 21:08:12,915 [I] Table x_portal_user does not exist in database ranger
2026-02-03 21:08:12,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-02-03 21:08:13,238 [I] Table x_policy_ref_group does not exist in database ranger
2026-02-03 21:08:13,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-02-03 21:08:13,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-02-03 21:08:13,891 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-02-03 21:08:13,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-02-03 21:08:14.808 UTC [12076] WARNING: there is no transaction in progress
2026-02-03 21:08:14.820 UTC [12076] WARNING: there is no transaction in progress
2026-02-03 21:08:14.834 UTC [12076] WARNING: there is no transaction in progress
2026-02-03 21:08:14.851 UTC [12076] WARNING: there is no transaction in progress
2026-02-03 21:08:14.863 UTC [12076] WARNING: there is no transaction in progress
2026-02-03 21:08:14.875 UTC [12076] WARNING: there is no transaction in progress
2026-02-03 21:08:14.925 UTC [12076] WARNING: there is no transaction in progress
2026-02-03 21:08:14.932 UTC [12076] WARNING: there is no transaction in progress
2026-02-03 21:08:14.942 UTC [12076] WARNING: there is no transaction in progress
2026-02-03 21:08:14.952 UTC [12076] WARNING: there is no transaction in progress
2026-02-03 21:08:14.962 UTC [12076] WARNING: there is no transaction in progress
2026-02-03 21:08:14.973 UTC [12076] WARNING: there is no transaction in progress
2026-02-03 21:08:14.982 UTC [12076] WARNING: there is no transaction in progress
2026-02-03 21:08:14.987 UTC [12076] WARNING: there is no transaction in progress
2026-02-03 21:08:14.991 UTC [12076] WARNING: there is no transaction in progress
2026-02-03 21:08:14.994 UTC [12076] WARNING: there is no transaction in progress
2026-02-03 21:08:15,429 [I] ranger_core_db_postgres.sql file imported successfully
2026-02-03 21:08:15,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-02-03 21:08:15,758 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-02-03 21:08:15,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-02-03 21:08:16,066 [I] Table x_portal_user already exists in database 'ranger'
2026-02-03 21:08:16,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-02-03 21:08:16,409 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-02-03 21:08:16,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-02-03 21:08:16,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-02-03 21:08:17,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j889.c.gcp-upstream.internal';"
2026-02-03 21:08:17,355 [I] CORE_DB_SCHEMA import status has been updated
2026-02-03 21:08:17,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-02-03 21:08:17,684 [I] DB_PATCHES have already been applied
I20260203 21:08:17.690166 410 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-admin:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3aGl8N/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260203 21:08:17.690241 410 mini_ranger.cc:193] Using host: 127.0.102.148
I20260203 21:08:17.693126 410 mini_ranger.cc:227] Ranger admin URL: http://127.0.102.148:40649
Feb 03, 2026 9:08:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Feb 03, 2026 9:08:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Feb 03, 2026 9:08:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-admin/ews/webapp
Feb 03, 2026 9:08:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Feb 03, 2026 9:08:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-admin/ews/webapp] .....
Feb 03, 2026 9:08:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-admin/ews/webapp].
Feb 03, 2026 9:08:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260203 21:08:19.743574 410 mini_ranger.cc:160] Time spent starting Ranger: real 10.040s user 0.001s sys 0.009s
Feb 03, 2026 9:08:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Feb 03, 2026 9:08:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
Feb 03, 2026 9:08:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Feb 03, 2026 9:08:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Feb 03, 2026 9:08:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Feb 03, 2026 9:08:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Feb 03, 2026 9:08:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Feb 03, 2026 9:08:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Feb 03, 2026 9:08:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Feb 03, 2026 9:08:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Feb 03, 2026 9:08:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Feb 03, 2026 9:08:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Feb 03, 2026 9:08:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Feb 03, 2026 9:08:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Feb 03, 2026 9:08:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Feb 03, 2026 9:08:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Feb 03, 2026 9:08:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Feb 03, 2026 9:08:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260203 21:08:41.963755 410 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.102.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.102.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.102.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/rangerkms_127.0.102.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.102.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/rangerkms_127.0.102.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.102.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.102.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/HTTP_127.0.102.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.102.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/HTTP_127.0.102.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Feb 03 21:08:42 dist-test-slave-j889 krb5kdc[11850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152922, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260203 21:08:42.063891 410 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260203 21:08:42.074322 410 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260203 21:08:42.194891 410 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260203 21:08:42.195016 410 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-kms
I20260203 21:08:42.195055 410 mini_ranger_kms.cc:227] Using postgres at 127.0.102.148:43925
2026-02-03 21:08:42,421 [I] DB FLAVOR :POSTGRES
2026-02-03 21:08:42,421 [I] --------- Verifying Ranger DB connection ---------
2026-02-03 21:08:42,422 [I] Checking connection
2026-02-03 21:08:42,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-02-03 21:08:42,791 [I] connection success
2026-02-03 21:08:42,791 [I] --------- Verifying Ranger DB tables ---------
2026-02-03 21:08:42,791 [I] Verifying table ranger_masterkey in database rangerkms
2026-02-03 21:08:42,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-02-03 21:08:43,167 [I] Table ranger_masterkey does not exist in database rangerkms
2026-02-03 21:08:43,167 [I] --------- Importing Ranger Core DB Schema ---------
2026-02-03 21:08:43,167 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-02-03 21:08:43,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:43925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-02-03 21:08:43,471 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260203 21:08:43.961710 410 mini_ranger_kms.cc:310] Created kms service
I20260203 21:08:44.212936 410 mini_ranger_kms.cc:326] Created kudu user
I20260203 21:08:44.262542 410 mini_ranger_kms.cc:343] Created rangerkms user
Feb 03 21:08:44 dist-test-slave-j889 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152922, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.102.148@KRBTEST.COM
I20260203 21:08:44.661226 410 mini_ranger_kms.cc:384] Added ranger policy
I20260203 21:08:44.661506 410 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-kms:/tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260203 21:08:44.661556 410 mini_ranger_kms.cc:240] Using host: 127.0.102.148
I20260203 21:08:44.664808 410 mini_ranger_kms.cc:276] Ranger KMS PID: 12268
I20260203 21:08:44.664919 410 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.102.148:60881
Feb 03, 2026 9:08:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Feb 03, 2026 9:08:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Feb 03, 2026 9:08:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Feb 03, 2026 9:08:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-kms/ews/webapp] .....
Feb 03, 2026 9:08:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-kms/ews/webapp].
Feb 03, 2026 9:08:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260203 21:08:46.666657 410 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.603s user 0.023s sys 0.184s
I20260203 21:08:46.666800 410 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260203 21:08:46.666886 410 mini_ranger_kms.cc:401] 127.0.102.148:60881/kms/v1/keys
Feb 03 21:08:46 dist-test-slave-j889 krb5kdc[11850](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1770152926, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.102.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Feb 03 21:08:47 dist-test-slave-j889 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152926, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.102.148@KRBTEST.COM for HTTP/127.0.102.148@KRBTEST.COM
Feb 03 21:08:47 dist-test-slave-j889 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152926, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.102.148@KRBTEST.COM for HTTP/127.0.102.148@KRBTEST.COM
Feb 03 21:08:49 dist-test-slave-j889 krb5kdc[11850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:08:49.993181 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:38967
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:38967
--ranger_config_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.102.148:60881/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:08:50.098397 12321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:08:50.098636 12321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:08:50.098707 12321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:08:50.101924 12321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:08:50.101989 12321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:08:50.102020 12321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:08:50.102044 12321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:08:50.102075 12321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:08:50.106318 12321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.102.148:60881/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:38967
--ranger_config_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:38967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:08:50.107367 12321 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:08:50.108180 12321 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:08:50.113867 12326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:08:50.113970 12329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:08:50.113867 12327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:08:50.114148 12321 server_base.cc:1060] running on GCE node
I20260203 21:08:50.114701 12321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:08:50.115609 12321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:08:50.116768 12321 hybrid_clock.cc:648] HybridClock initialized: now 1770152930116754 us; error 35 us; skew 500 ppm
Feb 03 21:08:50 dist-test-slave-j889 krb5kdc[11850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:08:50.119549 12321 init.cc:377] Logged in from keytab as kudu/127.0.102.190@KRBTEST.COM (short username kudu)
I20260203 21:08:50.120875 12321 webserver.cc:492] Webserver started at http://127.0.102.190:43687/ using document root <none> and password file <none>
I20260203 21:08:50.121402 12321 fs_manager.cc:362] Metadata directory not provided
I20260203 21:08:50.121450 12321 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:08:50.121625 12321 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 03 21:08:50 dist-test-slave-j889 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for HTTP/127.0.102.148@KRBTEST.COM
I20260203 21:08:50.176317 12321 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "df338f80ca5d45a48a179bfaad6cafdb"
format_stamp: "Formatted at 2026-02-03 21:08:50 on dist-test-slave-j889"
server_key: "d21bf86b0f1d60f16304f764088280c0"
server_key_iv: "f819be7c2bdca5078f95ddf7b0df7416"
server_key_version: "kuduclusterkey@0"
I20260203 21:08:50.176901 12321 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "df338f80ca5d45a48a179bfaad6cafdb"
format_stamp: "Formatted at 2026-02-03 21:08:50 on dist-test-slave-j889"
server_key: "d21bf86b0f1d60f16304f764088280c0"
server_key_iv: "f819be7c2bdca5078f95ddf7b0df7416"
server_key_version: "kuduclusterkey@0"
I20260203 21:08:50.180397 12321 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260203 21:08:50.194074 12336 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:50.195192 12321 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260203 21:08:50.195326 12321 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "df338f80ca5d45a48a179bfaad6cafdb"
format_stamp: "Formatted at 2026-02-03 21:08:50 on dist-test-slave-j889"
server_key: "d21bf86b0f1d60f16304f764088280c0"
server_key_iv: "f819be7c2bdca5078f95ddf7b0df7416"
server_key_version: "kuduclusterkey@0"
I20260203 21:08:50.195433 12321 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:08:50.215656 12321 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:08:50.221176 12321 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:08:50.221349 12321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:08:50.229444 12321 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:38967
I20260203 21:08:50.229480 12388 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:38967 every 8 connection(s)
I20260203 21:08:50.230440 12321 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:08:50.233798 12389 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:08:50.238749 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 12321
I20260203 21:08:50.238886 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:08:50.239135 410 external_mini_cluster.cc:1468] Setting key f831d24125374adb492edd4e22a8aaea
I20260203 21:08:50.240286 12389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb: Bootstrap starting.
I20260203 21:08:50.242858 12389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb: Neither blocks nor log segments found. Creating new log.
I20260203 21:08:50.243709 12389 log.cc:826] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb: Log is configured to *not* fsync() on all Append() calls
I20260203 21:08:50.245800 12389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb: No bootstrap required, opened a new log
Feb 03 21:08:50 dist-test-slave-j889 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:08:50.249519 12389 raft_consensus.cc:359] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df338f80ca5d45a48a179bfaad6cafdb" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 38967 } }
I20260203 21:08:50.249790 12389 raft_consensus.cc:385] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:08:50.249862 12389 raft_consensus.cc:740] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df338f80ca5d45a48a179bfaad6cafdb, State: Initialized, Role: FOLLOWER
I20260203 21:08:50.250391 12389 consensus_queue.cc:260] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df338f80ca5d45a48a179bfaad6cafdb" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 38967 } }
I20260203 21:08:50.250573 12389 raft_consensus.cc:399] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:08:50.250661 12389 raft_consensus.cc:493] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:08:50.250778 12389 raft_consensus.cc:3060] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:08:50.251573 12392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:50.240429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.0.102.190:38967)
0203 21:08:50.240840 (+ 411us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:50.240849 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:50.240882 (+ 33us) server_negotiation.cc:408] Connection header received
0203 21:08:50.241416 (+ 534us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:50.241431 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:50.241711 (+ 280us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:50.242007 (+ 296us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:08:50.242871 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:50.243630 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:50.244392 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:50.244719 (+ 327us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:50.246898 (+ 2179us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:08:50.246918 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:08:50.246931 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:08:50.246972 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:08:50.248920 (+ 1948us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:50.249378 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:50.249384 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:50.249390 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:50.249473 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:50.249708 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:50.249712 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:50.249714 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:50.250155 (+ 441us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:08:50.250371 (+ 216us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:50.250618 (+ 247us) server_negotiation.cc:300] Negotiation successful
0203 21:08:50.250838 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":149,"threads_started":1}
I20260203 21:08:50.251588 12389 raft_consensus.cc:515] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df338f80ca5d45a48a179bfaad6cafdb" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 38967 } }
I20260203 21:08:50.251966 12389 leader_election.cc:304] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df338f80ca5d45a48a179bfaad6cafdb; no voters:
I20260203 21:08:50.252251 12389 leader_election.cc:290] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:08:50.252358 12394 raft_consensus.cc:2804] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:08:50.252552 12394 raft_consensus.cc:697] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [term 1 LEADER]: Becoming Leader. State: Replica: df338f80ca5d45a48a179bfaad6cafdb, State: Running, Role: LEADER
I20260203 21:08:50.252887 12394 consensus_queue.cc:237] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df338f80ca5d45a48a179bfaad6cafdb" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 38967 } }
I20260203 21:08:50.253422 12389 sys_catalog.cc:565] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:08:50.254317 12394 sys_catalog.cc:455] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df338f80ca5d45a48a179bfaad6cafdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df338f80ca5d45a48a179bfaad6cafdb" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 38967 } } }
I20260203 21:08:50.254495 12394 sys_catalog.cc:458] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [sys.catalog]: This master's current role is: LEADER
I20260203 21:08:50.255214 12396 sys_catalog.cc:455] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader df338f80ca5d45a48a179bfaad6cafdb. Latest consensus state: current_term: 1 leader_uuid: "df338f80ca5d45a48a179bfaad6cafdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df338f80ca5d45a48a179bfaad6cafdb" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 38967 } } }
I20260203 21:08:50.255364 12396 sys_catalog.cc:458] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb [sys.catalog]: This master's current role is: LEADER
I20260203 21:08:50.255520 12403 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:08:50.258634 12403 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:08:50.259430 12389 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260203 21:08:50.274897 12403 catalog_manager.cc:1357] Generated new cluster ID: e5579b761b454723b3db0cd74a8a4c15
I20260203 21:08:50.274993 12403 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:08:50.298919 12403 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:08:50.300388 12403 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:08:50.314546 12403 catalog_manager.cc:6027] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb: Generated new TSK 0
I20260203 21:08:50.315629 12403 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:08:51.100589 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.102.148:60881/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:38967
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:08:51.232318 12435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:08:51.232545 12435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:08:51.232663 12435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:08:51.236114 12435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:08:51.236202 12435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:08:51.236330 12435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:08:51.240985 12435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.102.148:60881/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:38967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:08:51.242403 12435 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:08:51.243606 12435 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:08:51.251917 12442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:08:51.251956 12445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:08:51.251917 12443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:08:51.252250 12435 server_base.cc:1060] running on GCE node
I20260203 21:08:51.252622 12435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:08:51.253209 12435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:08:51.254376 12435 hybrid_clock.cc:648] HybridClock initialized: now 1770152931254351 us; error 45 us; skew 500 ppm
Feb 03 21:08:51 dist-test-slave-j889 krb5kdc[11850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:08:51.257077 12435 init.cc:377] Logged in from keytab as kudu/127.0.102.129@KRBTEST.COM (short username kudu)
I20260203 21:08:51.258198 12435 webserver.cc:492] Webserver started at http://127.0.102.129:33367/ using document root <none> and password file <none>
I20260203 21:08:51.258734 12435 fs_manager.cc:362] Metadata directory not provided
I20260203 21:08:51.258785 12435 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:08:51.258972 12435 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 03 21:08:51 dist-test-slave-j889 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for HTTP/127.0.102.148@KRBTEST.COM
I20260203 21:08:51.274168 12435 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "97f92db4b4fb41c99dcab8af900163cb"
format_stamp: "Formatted at 2026-02-03 21:08:51 on dist-test-slave-j889"
server_key: "6ba16b49f1bae46ec0fdb1bf48f4c31f"
server_key_iv: "25c8a207a951f376dd00b60dffef6c49"
server_key_version: "kuduclusterkey@0"
I20260203 21:08:51.274848 12435 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "97f92db4b4fb41c99dcab8af900163cb"
format_stamp: "Formatted at 2026-02-03 21:08:51 on dist-test-slave-j889"
server_key: "6ba16b49f1bae46ec0fdb1bf48f4c31f"
server_key_iv: "25c8a207a951f376dd00b60dffef6c49"
server_key_version: "kuduclusterkey@0"
I20260203 21:08:51.282649 12435 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260203 21:08:51.311609 12452 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:51.313227 12435 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.003s sys 0.000s
I20260203 21:08:51.313377 12435 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "97f92db4b4fb41c99dcab8af900163cb"
format_stamp: "Formatted at 2026-02-03 21:08:51 on dist-test-slave-j889"
server_key: "6ba16b49f1bae46ec0fdb1bf48f4c31f"
server_key_iv: "25c8a207a951f376dd00b60dffef6c49"
server_key_version: "kuduclusterkey@0"
I20260203 21:08:51.313512 12435 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:08:51.337574 12435 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:08:51.341223 12435 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:08:51.341442 12435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:08:51.342155 12435 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:08:51.343303 12435 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:08:51.343375 12435 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:51.343442 12435 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:08:51.343478 12435 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:51.359691 12435 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:37759
I20260203 21:08:51.362133 12567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:37759 every 8 connection(s)
I20260203 21:08:51.363550 12435 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:08:51.368521 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 12435
I20260203 21:08:51.368750 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:08:51.369020 410 external_mini_cluster.cc:1468] Setting key 418b4163db90ce44ead79b9562dee935
Feb 03 21:08:51 dist-test-slave-j889 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.102.130@KRBTEST.COM; defaulting to no policy
I20260203 21:08:51.382020 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:51.367947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:36853 (local address 127.0.102.190:38967)
0203 21:08:51.368186 (+ 239us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:51.368192 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:51.369582 (+ 1390us) server_negotiation.cc:408] Connection header received
0203 21:08:51.369630 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:51.369633 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:51.369691 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:51.369781 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:08:51.371506 (+ 1725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:51.372311 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:51.373363 (+ 1052us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:51.373554 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:51.376740 (+ 3186us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:08:51.376763 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:08:51.376767 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:08:51.376798 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:08:51.379503 (+ 2705us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:51.380195 (+ 692us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:51.380199 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:51.380202 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:51.380266 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:51.380682 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:51.380686 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:51.380688 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:51.380902 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:08:51.381005 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:51.381666 (+ 661us) server_negotiation.cc:300] Negotiation successful
0203 21:08:51.381807 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260203 21:08:51.382705 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:51.366849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:38967 (local address 127.0.102.129:36853)
0203 21:08:51.368190 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:51.368237 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0203 21:08:51.369268 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:08:51.370120 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:08:51.370132 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:08:51.370502 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:08:51.371317 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:51.371337 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:51.372469 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:08:51.372474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:08:51.373116 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:51.373128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:51.373354 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:51.375004 (+ 1650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:08:51.375047 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:08:51.376512 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:08:51.379667 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:08:51.379675 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:08:51.379690 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:08:51.380057 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:08:51.380386 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:08:51.380390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:08:51.380392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:08:51.380542 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:08:51.381081 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:08:51.381089 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:08:51.381405 (+ 316us) client_negotiation.cc:769] Sending connection context
0203 21:08:51.381619 (+ 214us) client_negotiation.cc:240] Negotiation successful
0203 21:08:51.381886 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1106,"thread_start_us":153,"threads_started":1}
I20260203 21:08:51.383898 12568 heartbeater.cc:344] Connected to a master server at 127.0.102.190:38967
I20260203 21:08:51.384156 12568 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.130@KRBTEST.COM" created.
I20260203 21:08:51.384763 12568 heartbeater.cc:507] Master 127.0.102.190:38967 requested a full tablet report, sending...
I20260203 21:08:51.386428 12353 ts_manager.cc:194] Registered new tserver with Master: 97f92db4b4fb41c99dcab8af900163cb (127.0.102.129:37759)
I20260203 21:08:51.387674 12353 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:36853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.130@KRBTEST.COM" created.
Feb 03 21:08:51 dist-test-slave-j889 krb5kdc[11850](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1770152931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:08:51.428088 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.102.148:60881/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:38967
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:08:51.568764 12576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:08:51.569063 12576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:08:51.569170 12576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:08:51.573886 12576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:08:51.573973 12576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:08:51.574102 12576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:08:51.580132 12576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.102.148:60881/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:38967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:08:51.581606 12576 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:08:51.582679 12576 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:08:51.590632 12582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:08:51.590637 12583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:08:51.591274 12585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:08:51.591718 12576 server_base.cc:1060] running on GCE node
I20260203 21:08:51.592132 12576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:08:51.592784 12576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:08:51.593963 12576 hybrid_clock.cc:648] HybridClock initialized: now 1770152931593948 us; error 32 us; skew 500 ppm
Feb 03 21:08:51 dist-test-slave-j889 krb5kdc[11850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:08:51.597424 12576 init.cc:377] Logged in from keytab as kudu/127.0.102.130@KRBTEST.COM (short username kudu)
I20260203 21:08:51.598954 12576 webserver.cc:492] Webserver started at http://127.0.102.130:36541/ using document root <none> and password file <none>
I20260203 21:08:51.599706 12576 fs_manager.cc:362] Metadata directory not provided
I20260203 21:08:51.599778 12576 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:08:51.600140 12576 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 03 21:08:51 dist-test-slave-j889 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for HTTP/127.0.102.148@KRBTEST.COM
I20260203 21:08:51.614159 12576 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "0cf7631ece49449592b917230a46b603"
format_stamp: "Formatted at 2026-02-03 21:08:51 on dist-test-slave-j889"
server_key: "e65ba71b490d2f23efec45da3d3356e0"
server_key_iv: "9b660b1c4a4ab4280b188602b9d011e0"
server_key_version: "kuduclusterkey@0"
I20260203 21:08:51.614817 12576 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "0cf7631ece49449592b917230a46b603"
format_stamp: "Formatted at 2026-02-03 21:08:51 on dist-test-slave-j889"
server_key: "e65ba71b490d2f23efec45da3d3356e0"
server_key_iv: "9b660b1c4a4ab4280b188602b9d011e0"
server_key_version: "kuduclusterkey@0"
I20260203 21:08:51.619319 12576 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260203 21:08:51.635620 12592 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:51.637270 12576 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260203 21:08:51.637537 12576 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "0cf7631ece49449592b917230a46b603"
format_stamp: "Formatted at 2026-02-03 21:08:51 on dist-test-slave-j889"
server_key: "e65ba71b490d2f23efec45da3d3356e0"
server_key_iv: "9b660b1c4a4ab4280b188602b9d011e0"
server_key_version: "kuduclusterkey@0"
I20260203 21:08:51.637735 12576 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:08:51.691879 12576 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:08:51.694846 12576 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:08:51.695043 12576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:08:51.695701 12576 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:08:51.696828 12576 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:08:51.696890 12576 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:51.696944 12576 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:08:51.696974 12576 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:51.708053 12576 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:41401
I20260203 21:08:51.709925 12576 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:08:51.714385 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 12576
I20260203 21:08:51.714519 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:08:51.714793 410 external_mini_cluster.cc:1468] Setting key cc718d3163270509c5c66ff017197cca
I20260203 21:08:51.722321 12705 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:41401 every 8 connection(s)
Feb 03 21:08:51 dist-test-slave-j889 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:08:51.737979 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:51.720870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.130:33989 (local address 127.0.102.190:38967)
0203 21:08:51.721030 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:51.721034 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:51.723662 (+ 2628us) server_negotiation.cc:408] Connection header received
0203 21:08:51.724880 (+ 1218us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:51.724885 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:51.724943 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:51.724992 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:08:51.726726 (+ 1734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:51.727601 (+ 875us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:51.728559 (+ 958us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:51.728792 (+ 233us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:51.731911 (+ 3119us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:08:51.731936 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:08:51.731940 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:08:51.731970 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:08:51.733560 (+ 1590us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:51.735988 (+ 2428us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:51.735993 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:51.735995 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:51.736060 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:51.736505 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:51.736510 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:51.736512 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:51.736771 (+ 259us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:08:51.736873 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:51.737660 (+ 787us) server_negotiation.cc:300] Negotiation successful
0203 21:08:51.737802 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.0.102.131@KRBTEST.COM; defaulting to no policy
I20260203 21:08:51.739339 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:51.721271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:38967 (local address 127.0.102.130:33989)
0203 21:08:51.723507 (+ 2236us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:51.723552 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0203 21:08:51.724600 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:08:51.725422 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:08:51.725434 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:08:51.725782 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:08:51.726552 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:51.726568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:51.727750 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:08:51.727753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:08:51.728427 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:51.728437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:51.729035 (+ 598us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:51.730175 (+ 1140us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:08:51.730211 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:08:51.731737 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:08:51.733691 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:08:51.733697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:08:51.733710 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:08:51.734018 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:08:51.736191 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:08:51.736195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:08:51.736197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:08:51.736365 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:08:51.736947 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:08:51.736955 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:08:51.737297 (+ 342us) client_negotiation.cc:769] Sending connection context
0203 21:08:51.738426 (+ 1129us) client_negotiation.cc:240] Negotiation successful
0203 21:08:51.738665 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2014,"thread_start_us":202,"threads_started":1}
I20260203 21:08:51.740459 12706 heartbeater.cc:344] Connected to a master server at 127.0.102.190:38967
I20260203 21:08:51.740738 12706 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.131@KRBTEST.COM" created.
I20260203 21:08:51.741367 12706 heartbeater.cc:507] Master 127.0.102.190:38967 requested a full tablet report, sending...
I20260203 21:08:51.742358 12353 ts_manager.cc:194] Registered new tserver with Master: 0cf7631ece49449592b917230a46b603 (127.0.102.130:41401)
I20260203 21:08:51.742944 12353 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.130@KRBTEST.COM'} at 127.0.102.130:33989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.131@KRBTEST.COM" created.
Feb 03 21:08:51 dist-test-slave-j889 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for HTTP/127.0.102.148@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:08:51.784149 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.102.148:60881/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.131:0
--local_ip_for_outbound_sockets=127.0.102.131
--webserver_interface=127.0.102.131
--webserver_port=0
--tserver_master_addrs=127.0.102.190:38967
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:08:51.907060 12713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:08:51.907282 12713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:08:51.907341 12713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:08:51.911309 12713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:08:51.911389 12713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:08:51.911500 12713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.131
I20260203 21:08:51.917495 12713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.102.148:60881/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.102.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:38967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.131
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:08:51.918908 12713 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:08:51.919973 12713 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:08:51.927897 12719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:08:51.928146 12718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:08:51.929483 12721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:08:51.929761 12713 server_base.cc:1060] running on GCE node
I20260203 21:08:51.930322 12713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:08:51.931144 12713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:08:51.932330 12713 hybrid_clock.cc:648] HybridClock initialized: now 1770152931932309 us; error 37 us; skew 500 ppm
Feb 03 21:08:51 dist-test-slave-j889 krb5kdc[11850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:08:51.935683 12713 init.cc:377] Logged in from keytab as kudu/127.0.102.131@KRBTEST.COM (short username kudu)
I20260203 21:08:51.937009 12713 webserver.cc:492] Webserver started at http://127.0.102.131:36213/ using document root <none> and password file <none>
I20260203 21:08:51.937677 12713 fs_manager.cc:362] Metadata directory not provided
I20260203 21:08:51.937743 12713 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:08:51.937947 12713 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 03 21:08:51 dist-test-slave-j889 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for HTTP/127.0.102.148@KRBTEST.COM
I20260203 21:08:51.956990 12713 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/data/instance:
uuid: "cb36a062854740db88a0f4165da05506"
format_stamp: "Formatted at 2026-02-03 21:08:51 on dist-test-slave-j889"
server_key: "fac517da19c1824507254ecb910ba1e1"
server_key_iv: "db769374d5645be0c10d6ef105484bdc"
server_key_version: "kuduclusterkey@0"
I20260203 21:08:51.957675 12713 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/wal/instance:
uuid: "cb36a062854740db88a0f4165da05506"
format_stamp: "Formatted at 2026-02-03 21:08:51 on dist-test-slave-j889"
server_key: "fac517da19c1824507254ecb910ba1e1"
server_key_iv: "db769374d5645be0c10d6ef105484bdc"
server_key_version: "kuduclusterkey@0"
I20260203 21:08:51.965354 12713 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260203 21:08:51.985590 12728 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:51.986896 12713 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260203 21:08:51.987020 12713 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/wal
uuid: "cb36a062854740db88a0f4165da05506"
format_stamp: "Formatted at 2026-02-03 21:08:51 on dist-test-slave-j889"
server_key: "fac517da19c1824507254ecb910ba1e1"
server_key_iv: "db769374d5645be0c10d6ef105484bdc"
server_key_version: "kuduclusterkey@0"
I20260203 21:08:51.987129 12713 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:08:52.002413 12713 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:08:52.005755 12713 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:08:52.005950 12713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:08:52.006628 12713 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:08:52.007649 12713 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:08:52.007710 12713 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:08:52.007757 12713 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:08:52.007783 12713 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Feb 03 21:08:52 dist-test-slave-j889 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for HTTP/127.0.102.148@KRBTEST.COM
I20260203 21:08:52.018751 12713 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.131:45479
I20260203 21:08:52.019827 12713 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/data/info.pb
I20260203 21:08:52.023087 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 12713
I20260203 21:08:52.023227 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/wal/instance
I20260203 21:08:52.023469 410 external_mini_cluster.cc:1468] Setting key d0ef3df033eba86f2d0f64e1bb218bcb
I20260203 21:08:52.025864 12842 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.131:45479 every 8 connection(s)
Feb 03 21:08:52 dist-test-slave-j889 krb5kdc[11850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.131@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:08:52.043085 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:52.027827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:58449 (local address 127.0.102.190:38967)
0203 21:08:52.031562 (+ 3735us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:52.031566 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:52.031582 (+ 16us) server_negotiation.cc:408] Connection header received
0203 21:08:52.031624 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:52.031628 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:52.031680 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:52.031764 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:08:52.033336 (+ 1572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.034111 (+ 775us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:52.035119 (+ 1008us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.035337 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:52.038334 (+ 2997us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:08:52.038353 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:08:52.038357 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:08:52.038403 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:08:52.040255 (+ 1852us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:52.041257 (+ 1002us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:52.041261 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:52.041264 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:52.041321 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:52.041727 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:52.041732 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:52.041734 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:52.041932 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:08:52.042018 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:52.042748 (+ 730us) server_negotiation.cc:300] Negotiation successful
0203 21:08:52.042890 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3634}
I20260203 21:08:52.044502 12845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:52.027650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:38967 (local address 127.0.102.131:58449)
0203 21:08:52.028773 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:52.028818 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0203 21:08:52.029752 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:08:52.031981 (+ 2229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:08:52.031994 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:08:52.032347 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:08:52.033164 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:52.033182 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.034280 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:08:52.034284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:08:52.034953 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:52.034966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.035623 (+ 657us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:52.036282 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:08:52.036329 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:08:52.038135 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:08:52.040754 (+ 2619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:08:52.040761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:08:52.040776 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:08:52.041122 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:08:52.041443 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:08:52.041447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:08:52.041450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:08:52.041605 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:08:52.042087 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:08:52.042095 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:08:52.042426 (+ 331us) client_negotiation.cc:769] Sending connection context
0203 21:08:52.043328 (+ 902us) client_negotiation.cc:240] Negotiation successful
0203 21:08:52.043625 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"thread_start_us":137,"threads_started":1}
I20260203 21:08:52.045672 12843 heartbeater.cc:344] Connected to a master server at 127.0.102.190:38967
I20260203 21:08:52.045953 12843 heartbeater.cc:461] Registering TS with master...
I20260203 21:08:52.046628 12843 heartbeater.cc:507] Master 127.0.102.190:38967 requested a full tablet report, sending...
I20260203 21:08:52.047693 12353 ts_manager.cc:194] Registered new tserver with Master: cb36a062854740db88a0f4165da05506 (127.0.102.131:45479)
I20260203 21:08:52.048365 12353 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.131@KRBTEST.COM'} at 127.0.102.131:58449
I20260203 21:08:52.049897 410 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260203 21:08:52.059675 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:52.052103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48316 (local address 127.0.102.190:38967)
0203 21:08:52.052227 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:52.052233 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:52.052719 (+ 486us) server_negotiation.cc:408] Connection header received
0203 21:08:52.052838 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:52.052842 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:52.052890 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:52.052962 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:08:52.053820 (+ 858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.054324 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:52.055138 (+ 814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.055327 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:52.056621 (+ 1294us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:08:52.056671 (+ 50us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:08:52.056674 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:08:52.056702 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:08:52.058120 (+ 1418us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:52.058541 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:52.058545 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:52.058546 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:52.058587 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:08:52.058801 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:08:52.058805 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:08:52.058807 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:08:52.058991 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:08:52.059132 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:52.059372 (+ 240us) server_negotiation.cc:300] Negotiation successful
0203 21:08:52.059508 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260203 21:08:52.062302 12353 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260203 21:08:52.065454 12353 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260203 21:08:52.080979 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:52.074648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:41401 (local address 127.0.0.1:44986)
0203 21:08:52.076411 (+ 1763us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:52.076425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:08:52.076508 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:08:52.077176 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:08:52.077182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:08:52.077205 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:08:52.077498 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:52.077507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.078911 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:08:52.078915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:08:52.080459 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:52.080468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.080567 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:52.080618 (+ 51us) client_negotiation.cc:769] Sending connection context
0203 21:08:52.080736 (+ 118us) client_negotiation.cc:240] Negotiation successful
0203 21:08:52.080820 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1689,"thread_start_us":99,"threads_started":1}
I20260203 21:08:52.081331 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:52.074440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.131:45479 (local address 127.0.0.1:60334)
0203 21:08:52.075706 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:52.075748 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0203 21:08:52.075883 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:08:52.077846 (+ 1963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:08:52.077849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:08:52.077867 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:08:52.078122 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:52.078131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.080017 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:08:52.080021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:08:52.080969 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:52.080980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.081090 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:52.081107 (+ 17us) client_negotiation.cc:769] Sending connection context
0203 21:08:52.081153 (+ 46us) client_negotiation.cc:240] Negotiation successful
0203 21:08:52.081208 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"thread_start_us":94,"threads_started":1}
I20260203 21:08:52.082211 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:52.074824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.0.102.130:41401)
0203 21:08:52.076761 (+ 1937us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:52.076768 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:52.076792 (+ 24us) server_negotiation.cc:408] Connection header received
0203 21:08:52.076875 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:52.076881 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:52.077030 (+ 149us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:52.077157 (+ 127us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:08:52.077640 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.078782 (+ 1142us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:52.081304 (+ 2522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.081823 (+ 519us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:52.081939 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:52.081997 (+ 58us) server_negotiation.cc:300] Negotiation successful
0203 21:08:52.082078 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1827,"thread_start_us":119,"threads_started":1}
I20260203 21:08:52.082815 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:52.076020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41992 (local address 127.0.102.129:37759)
0203 21:08:52.078128 (+ 2108us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:52.078135 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:52.078459 (+ 324us) server_negotiation.cc:408] Connection header received
0203 21:08:52.078541 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:52.078546 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:52.078691 (+ 145us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:52.078805 (+ 114us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:08:52.079335 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.080400 (+ 1065us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:52.081886 (+ 1486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.082467 (+ 581us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:52.082568 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:52.082622 (+ 54us) server_negotiation.cc:300] Negotiation successful
0203 21:08:52.082696 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2006,"thread_start_us":66,"threads_started":1}
I20260203 21:08:52.083686 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:52.075237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.0.102.131:45479)
0203 21:08:52.077480 (+ 2243us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:52.077485 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:52.077507 (+ 22us) server_negotiation.cc:408] Connection header received
0203 21:08:52.077587 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:52.077593 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:52.077729 (+ 136us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:52.077846 (+ 117us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:08:52.078258 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.079891 (+ 1633us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:52.081595 (+ 1704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.082062 (+ 467us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:52.083387 (+ 1325us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:52.083462 (+ 75us) server_negotiation.cc:300] Negotiation successful
0203 21:08:52.083549 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2137,"thread_start_us":125,"threads_started":1}
I20260203 21:08:52.081796 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:52.075666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:37759 (local address 127.0.0.1:41992)
0203 21:08:52.078279 (+ 2613us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:52.078293 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:08:52.078370 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:08:52.078936 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:08:52.078939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:08:52.078956 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:08:52.079211 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:52.079220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.080517 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:08:52.080520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:08:52.081472 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:52.081481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.081576 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:52.081590 (+ 14us) client_negotiation.cc:769] Sending connection context
0203 21:08:52.081634 (+ 44us) client_negotiation.cc:240] Negotiation successful
0203 21:08:52.081684 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2549,"thread_start_us":82,"threads_started":1}
I20260203 21:08:52.085198 12640 tablet_service.cc:1505] Processing CreateTablet for tablet 6e66720e83ab42169146e6cf10b80810 (DEFAULT_TABLE table=test-table [id=f4af432dc9be4693b2195b3a797e17da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:08:52.086130 12640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e66720e83ab42169146e6cf10b80810. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:08:52.086733 12768 tablet_service.cc:1505] Processing CreateTablet for tablet 6e66720e83ab42169146e6cf10b80810 (DEFAULT_TABLE table=test-table [id=f4af432dc9be4693b2195b3a797e17da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:08:52.087514 12768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e66720e83ab42169146e6cf10b80810. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:08:52.090632 12483 tablet_service.cc:1505] Processing CreateTablet for tablet 6e66720e83ab42169146e6cf10b80810 (DEFAULT_TABLE table=test-table [id=f4af432dc9be4693b2195b3a797e17da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260203 21:08:52.091557 12483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e66720e83ab42169146e6cf10b80810. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:08:52.094493 12862 tablet_bootstrap.cc:492] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506: Bootstrap starting.
I20260203 21:08:52.096724 12862 tablet_bootstrap.cc:654] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506: Neither blocks nor log segments found. Creating new log.
I20260203 21:08:52.097889 12862 log.cc:826] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506: Log is configured to *not* fsync() on all Append() calls
I20260203 21:08:52.099812 12862 tablet_bootstrap.cc:492] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506: No bootstrap required, opened a new log
I20260203 21:08:52.099992 12862 ts_tablet_manager.cc:1403] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260203 21:08:52.103035 12862 raft_consensus.cc:359] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb36a062854740db88a0f4165da05506" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 45479 } } peers { permanent_uuid: "0cf7631ece49449592b917230a46b603" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41401 } } peers { permanent_uuid: "97f92db4b4fb41c99dcab8af900163cb" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37759 } }
I20260203 21:08:52.103253 12862 raft_consensus.cc:385] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:08:52.103273 12864 tablet_bootstrap.cc:492] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603: Bootstrap starting.
I20260203 21:08:52.103312 12862 raft_consensus.cc:740] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb36a062854740db88a0f4165da05506, State: Initialized, Role: FOLLOWER
I20260203 21:08:52.104872 12862 consensus_queue.cc:260] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb36a062854740db88a0f4165da05506" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 45479 } } peers { permanent_uuid: "0cf7631ece49449592b917230a46b603" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41401 } } peers { permanent_uuid: "97f92db4b4fb41c99dcab8af900163cb" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37759 } }
I20260203 21:08:52.105427 12864 tablet_bootstrap.cc:654] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603: Neither blocks nor log segments found. Creating new log.
I20260203 21:08:52.106035 12862 ts_tablet_manager.cc:1434] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260203 21:08:52.106441 12843 heartbeater.cc:499] Master 127.0.102.190:38967 was elected leader, sending a full tablet report...
I20260203 21:08:52.106575 12864 log.cc:826] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603: Log is configured to *not* fsync() on all Append() calls
I20260203 21:08:52.106725 12865 tablet_bootstrap.cc:492] T 6e66720e83ab42169146e6cf10b80810 P 97f92db4b4fb41c99dcab8af900163cb: Bootstrap starting.
I20260203 21:08:52.108306 12864 tablet_bootstrap.cc:492] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603: No bootstrap required, opened a new log
I20260203 21:08:52.108496 12864 ts_tablet_manager.cc:1403] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260203 21:08:52.109119 12865 tablet_bootstrap.cc:654] T 6e66720e83ab42169146e6cf10b80810 P 97f92db4b4fb41c99dcab8af900163cb: Neither blocks nor log segments found. Creating new log.
I20260203 21:08:52.109936 12865 log.cc:826] T 6e66720e83ab42169146e6cf10b80810 P 97f92db4b4fb41c99dcab8af900163cb: Log is configured to *not* fsync() on all Append() calls
I20260203 21:08:52.111634 12864 raft_consensus.cc:359] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb36a062854740db88a0f4165da05506" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 45479 } } peers { permanent_uuid: "0cf7631ece49449592b917230a46b603" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41401 } } peers { permanent_uuid: "97f92db4b4fb41c99dcab8af900163cb" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37759 } }
I20260203 21:08:52.111842 12864 raft_consensus.cc:385] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:08:52.111889 12864 raft_consensus.cc:740] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf7631ece49449592b917230a46b603, State: Initialized, Role: FOLLOWER
I20260203 21:08:52.112350 12864 consensus_queue.cc:260] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb36a062854740db88a0f4165da05506" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 45479 } } peers { permanent_uuid: "0cf7631ece49449592b917230a46b603" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41401 } } peers { permanent_uuid: "97f92db4b4fb41c99dcab8af900163cb" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37759 } }
I20260203 21:08:52.113219 12864 ts_tablet_manager.cc:1434] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260203 21:08:52.113598 12706 heartbeater.cc:499] Master 127.0.102.190:38967 was elected leader, sending a full tablet report...
I20260203 21:08:52.118747 12866 raft_consensus.cc:493] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260203 21:08:52.119004 12866 raft_consensus.cc:515] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb36a062854740db88a0f4165da05506" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 45479 } } peers { permanent_uuid: "0cf7631ece49449592b917230a46b603" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41401 } } peers { permanent_uuid: "97f92db4b4fb41c99dcab8af900163cb" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37759 } }
I20260203 21:08:52.119427 12865 tablet_bootstrap.cc:492] T 6e66720e83ab42169146e6cf10b80810 P 97f92db4b4fb41c99dcab8af900163cb: No bootstrap required, opened a new log
I20260203 21:08:52.119655 12865 ts_tablet_manager.cc:1403] T 6e66720e83ab42169146e6cf10b80810 P 97f92db4b4fb41c99dcab8af900163cb: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260203 21:08:52.121126 12866 leader_election.cc:290] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf7631ece49449592b917230a46b603 (127.0.102.130:41401), 97f92db4b4fb41c99dcab8af900163cb (127.0.102.129:37759)
I20260203 21:08:52.124158 12845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:52.120998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:37759 (local address 127.0.102.131:44877)
0203 21:08:52.121141 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:52.121158 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0203 21:08:52.121250 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:08:52.121605 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:08:52.121609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:08:52.121629 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:08:52.121879 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:52.121886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.122892 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:08:52.122896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:08:52.123791 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:52.123800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.123902 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:52.123923 (+ 21us) client_negotiation.cc:769] Sending connection context
0203 21:08:52.123968 (+ 45us) client_negotiation.cc:240] Negotiation successful
0203 21:08:52.124029 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260203 21:08:52.124177 12865 raft_consensus.cc:359] T 6e66720e83ab42169146e6cf10b80810 P 97f92db4b4fb41c99dcab8af900163cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb36a062854740db88a0f4165da05506" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 45479 } } peers { permanent_uuid: "0cf7631ece49449592b917230a46b603" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41401 } } peers { permanent_uuid: "97f92db4b4fb41c99dcab8af900163cb" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37759 } }
I20260203 21:08:52.124392 12865 raft_consensus.cc:385] T 6e66720e83ab42169146e6cf10b80810 P 97f92db4b4fb41c99dcab8af900163cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:08:52.124442 12865 raft_consensus.cc:740] T 6e66720e83ab42169146e6cf10b80810 P 97f92db4b4fb41c99dcab8af900163cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f92db4b4fb41c99dcab8af900163cb, State: Initialized, Role: FOLLOWER
I20260203 21:08:52.124940 12865 consensus_queue.cc:260] T 6e66720e83ab42169146e6cf10b80810 P 97f92db4b4fb41c99dcab8af900163cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb36a062854740db88a0f4165da05506" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 45479 } } peers { permanent_uuid: "0cf7631ece49449592b917230a46b603" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41401 } } peers { permanent_uuid: "97f92db4b4fb41c99dcab8af900163cb" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37759 } }
I20260203 21:08:52.125072 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:52.120793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:44877 (local address 127.0.102.129:37759)
0203 21:08:52.120909 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:52.120913 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:52.121330 (+ 417us) server_negotiation.cc:408] Connection header received
0203 21:08:52.121377 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:52.121380 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:52.121438 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:52.121499 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:08:52.122013 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.122767 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:52.124310 (+ 1543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.124845 (+ 535us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:52.124880 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:52.124925 (+ 45us) server_negotiation.cc:300] Negotiation successful
0203 21:08:52.124974 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260203 21:08:52.125604 12522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e66720e83ab42169146e6cf10b80810" candidate_uuid: "cb36a062854740db88a0f4165da05506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f92db4b4fb41c99dcab8af900163cb" is_pre_election: true
I20260203 21:08:52.125841 12865 ts_tablet_manager.cc:1434] T 6e66720e83ab42169146e6cf10b80810 P 97f92db4b4fb41c99dcab8af900163cb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260203 21:08:52.125893 12522 raft_consensus.cc:2468] T 6e66720e83ab42169146e6cf10b80810 P 97f92db4b4fb41c99dcab8af900163cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb36a062854740db88a0f4165da05506 in term 0.
I20260203 21:08:52.125610 12568 heartbeater.cc:499] Master 127.0.102.190:38967 was elected leader, sending a full tablet report...
I20260203 21:08:52.126339 12729 leader_election.cc:304] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97f92db4b4fb41c99dcab8af900163cb, cb36a062854740db88a0f4165da05506; no voters:
I20260203 21:08:52.126575 12866 raft_consensus.cc:2804] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260203 21:08:52.126627 12866 raft_consensus.cc:493] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260203 21:08:52.126662 12866 raft_consensus.cc:3060] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:08:52.127715 12866 raft_consensus.cc:515] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb36a062854740db88a0f4165da05506" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 45479 } } peers { permanent_uuid: "0cf7631ece49449592b917230a46b603" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41401 } } peers { permanent_uuid: "97f92db4b4fb41c99dcab8af900163cb" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37759 } }
I20260203 21:08:52.128351 12866 leader_election.cc:290] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf7631ece49449592b917230a46b603 (127.0.102.130:41401), 97f92db4b4fb41c99dcab8af900163cb (127.0.102.129:37759)
I20260203 21:08:52.128412 12522 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e66720e83ab42169146e6cf10b80810" candidate_uuid: "cb36a062854740db88a0f4165da05506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f92db4b4fb41c99dcab8af900163cb"
I20260203 21:08:52.128515 12522 raft_consensus.cc:3060] T 6e66720e83ab42169146e6cf10b80810 P 97f92db4b4fb41c99dcab8af900163cb [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:08:52.129577 12522 raft_consensus.cc:2468] T 6e66720e83ab42169146e6cf10b80810 P 97f92db4b4fb41c99dcab8af900163cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb36a062854740db88a0f4165da05506 in term 1.
I20260203 21:08:52.129860 12729 leader_election.cc:304] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97f92db4b4fb41c99dcab8af900163cb, cb36a062854740db88a0f4165da05506; no voters:
I20260203 21:08:52.130154 12866 raft_consensus.cc:2804] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:08:52.130353 12866 raft_consensus.cc:697] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [term 1 LEADER]: Becoming Leader. State: Replica: cb36a062854740db88a0f4165da05506, State: Running, Role: LEADER
I20260203 21:08:52.130642 12866 consensus_queue.cc:237] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb36a062854740db88a0f4165da05506" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 45479 } } peers { permanent_uuid: "0cf7631ece49449592b917230a46b603" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41401 } } peers { permanent_uuid: "97f92db4b4fb41c99dcab8af900163cb" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37759 } }
I20260203 21:08:52.133281 12353 catalog_manager.cc:5654] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb36a062854740db88a0f4165da05506 (127.0.102.131). New cstate: current_term: 1 leader_uuid: "cb36a062854740db88a0f4165da05506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb36a062854740db88a0f4165da05506" member_type: VOTER last_known_addr { host: "127.0.102.131" port: 45479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cf7631ece49449592b917230a46b603" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97f92db4b4fb41c99dcab8af900163cb" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37759 } health_report { overall_health: UNKNOWN } } }
I20260203 21:08:52.134716 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:52.120998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.130:41401 (local address 127.0.102.131:53629)
0203 21:08:52.121382 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:52.121396 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0203 21:08:52.125811 (+ 4415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:08:52.126178 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:08:52.126181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:08:52.126199 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0203 21:08:52.126502 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:52.126511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.133355 (+ 6844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:08:52.133359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:08:52.134300 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:08:52.134310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.134437 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:52.134454 (+ 17us) client_negotiation.cc:769] Sending connection context
0203 21:08:52.134503 (+ 49us) client_negotiation.cc:240] Negotiation successful
0203 21:08:52.134564 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":34,"thread_start_us":120,"threads_started":1}
I20260203 21:08:52.135206 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:52.120800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.131:53629 (local address 127.0.102.130:41401)
0203 21:08:52.120909 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:52.120913 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:52.121471 (+ 558us) server_negotiation.cc:408] Connection header received
0203 21:08:52.125949 (+ 4478us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:52.125953 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:52.126005 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:52.126052 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0203 21:08:52.126630 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.127306 (+ 676us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:52.134436 (+ 7130us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.134936 (+ 500us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:52.134975 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:52.135029 (+ 54us) server_negotiation.cc:300] Negotiation successful
0203 21:08:52.135086 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260203 21:08:52.135742 12659 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e66720e83ab42169146e6cf10b80810" candidate_uuid: "cb36a062854740db88a0f4165da05506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf7631ece49449592b917230a46b603"
I20260203 21:08:52.135766 12660 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6e66720e83ab42169146e6cf10b80810" candidate_uuid: "cb36a062854740db88a0f4165da05506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf7631ece49449592b917230a46b603" is_pre_election: true
I20260203 21:08:52.135921 12659 raft_consensus.cc:3060] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603 [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:08:52.137101 12659 raft_consensus.cc:2468] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb36a062854740db88a0f4165da05506 in term 1.
I20260203 21:08:52.150620 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:52.147422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60342 (local address 127.0.102.131:45479)
0203 21:08:52.147536 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0203 21:08:52.147540 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:08:52.147554 (+ 14us) server_negotiation.cc:408] Connection header received
0203 21:08:52.147714 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:08:52.147717 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:08:52.147783 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:08:52.147872 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
0203 21:08:52.148331 (+ 459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.148916 (+ 585us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:08:52.149561 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:08:52.149721 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:08:52.149806 (+ 85us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0203 21:08:52.150228 (+ 422us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0203 21:08:52.150334 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0203 21:08:52.150453 (+ 119us) server_negotiation.cc:300] Negotiation successful
0203 21:08:52.150498 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260203 21:08:52.156790 12522 raft_consensus.cc:1275] T 6e66720e83ab42169146e6cf10b80810 P 97f92db4b4fb41c99dcab8af900163cb [term 1 FOLLOWER]: Refusing update from remote peer cb36a062854740db88a0f4165da05506: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:08:52.157572 12872 consensus_queue.cc:1048] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97f92db4b4fb41c99dcab8af900163cb" member_type: VOTER last_known_addr { host: "127.0.102.129" port: 37759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:08:52.159322 12659 raft_consensus.cc:1275] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603 [term 1 FOLLOWER]: Refusing update from remote peer cb36a062854740db88a0f4165da05506: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260203 21:08:52.159847 12872 consensus_queue.cc:1048] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf7631ece49449592b917230a46b603" member_type: VOTER last_known_addr { host: "127.0.102.130" port: 41401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260203 21:08:52.170550 12878 mvcc.cc:204] Tried to move back new op lower bound from 7250546410105008128 to 7250546410009624576. Current Snapshot: MvccSnapshot[applied={T|T < 7250546410105008128}]
I20260203 21:08:52.180897 12876 mvcc.cc:204] Tried to move back new op lower bound from 7250546410105008128 to 7250546410009624576. Current Snapshot: MvccSnapshot[applied={T|T < 7250546410105008128 or (T in {7250546410105008128})}]
I20260203 21:08:52.181687 12353 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:08:52.181838 12353 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260203 21:08:52.184266 12353 catalog_manager.cc:5941] T 00000000000000000000000000000000 P df338f80ca5d45a48a179bfaad6cafdb: Sending DeleteTablet for 3 replicas of tablet 6e66720e83ab42169146e6cf10b80810
I20260203 21:08:52.185194 12640 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e66720e83ab42169146e6cf10b80810 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:08:52 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:44986
I20260203 21:08:52.185995 12483 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e66720e83ab42169146e6cf10b80810 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:08:52 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:41992
I20260203 21:08:52.186659 12768 tablet_service.cc:1552] Processing DeleteTablet for tablet 6e66720e83ab42169146e6cf10b80810 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-02-03 21:08:52 UTC) from {username='kudu', principal='kudu/127.0.102.190@KRBTEST.COM'} at 127.0.0.1:60334
I20260203 21:08:52.187618 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 12435
I20260203 21:08:52.190378 12886 tablet_replica.cc:333] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506: stopping tablet replica
I20260203 21:08:52.190641 12886 raft_consensus.cc:2243] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [term 1 LEADER]: Raft consensus shutting down.
I20260203 21:08:52.191025 12886 raft_consensus.cc:2272] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:08:52.194221 12886 ts_tablet_manager.cc:1916] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:08:52.196964 12886 ts_tablet_manager.cc:1929] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:08:52.197067 12886 log.cc:1199] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-2/wal/wals/6e66720e83ab42169146e6cf10b80810
I20260203 21:08:52.197443 12886 ts_tablet_manager.cc:1950] T 6e66720e83ab42169146e6cf10b80810 P cb36a062854740db88a0f4165da05506: Deleting consensus metadata
I20260203 21:08:52.198756 12337 catalog_manager.cc:4985] TS cb36a062854740db88a0f4165da05506 (127.0.102.131:45479): tablet 6e66720e83ab42169146e6cf10b80810 (table test-table [id=f4af432dc9be4693b2195b3a797e17da]) successfully deleted
I20260203 21:08:52.199177 12884 tablet_replica.cc:333] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603: stopping tablet replica
I20260203 21:08:52.199476 12884 raft_consensus.cc:2243] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260203 21:08:52.199828 12884 raft_consensus.cc:2272] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260203 21:08:52.204569 12884 ts_tablet_manager.cc:1916] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260203 21:08:52.207073 12884 ts_tablet_manager.cc:1929] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260203 21:08:52.207171 12884 log.cc:1199] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603: Deleting WAL directory at /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1770152788143875-410-0/minicluster-data/ts-1/wal/wals/6e66720e83ab42169146e6cf10b80810
I20260203 21:08:52.207459 12884 ts_tablet_manager.cc:1950] T 6e66720e83ab42169146e6cf10b80810 P 0cf7631ece49449592b917230a46b603: Deleting consensus metadata
I20260203 21:08:52.208665 12338 catalog_manager.cc:4985] TS 0cf7631ece49449592b917230a46b603 (127.0.102.130:41401): tablet 6e66720e83ab42169146e6cf10b80810 (table test-table [id=f4af432dc9be4693b2195b3a797e17da]) successfully deleted
W20260203 21:08:52.208971 12337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.102.129:37759 (error 108)
I20260203 21:08:52.209905 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 12576
I20260203 21:08:52.211563 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:08:52.210996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.129:37759 (local address 127.0.0.1:42006)
0203 21:08:52.211389 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0203 21:08:52.211474 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.129:37759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287}
W20260203 21:08:52.211723 12337 catalog_manager.cc:4712] TS 97f92db4b4fb41c99dcab8af900163cb (127.0.102.129:37759): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e66720e83ab42169146e6cf10b80810: Network error: Client connection negotiation failed: client connection to 127.0.102.129:37759: connect: Connection refused (error 111)
I20260203 21:08:52.218411 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 12713
I20260203 21:08:52.224789 410 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task3aGl8N/build/debug/bin/kudu with pid 12321
I20260203 21:08:52.230556 410 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260203 21:08:52.565152 410 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260203 21:08:52.565290 410 mini_ranger.cc:66] Stopping Ranger...
I20260203 21:08:52.900177 410 mini_ranger.cc:68] Stopped Ranger
2026-02-03 21:08:52.900 UTC [11881] LOG: received smart shutdown request
2026-02-03 21:08:52.903 UTC [11881] LOG: background worker "logical replication launcher" (PID 11889) exited with exit code 1
2026-02-03 21:08:52.904 UTC [11884] LOG: shutting down
2026-02-03 21:08:52.905 UTC [11884] LOG: checkpoint starting: shutdown immediate
2026-02-03 21:08:53.750 UTC [11884] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.033 s, sync=0.807 s, total=0.847 s; sync files=1108, longest=0.005 s, average=0.001 s; distance=12074 kB, estimate=12074 kB; lsn=0/2045150, redo lsn=0/2045150
2026-02-03 21:08:53.757 UTC [11881] LOG: database system is shut down
2026-02-03T21:08:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51321 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Feb 03 21:08:53 dist-test-slave-j889 krb5kdc[12896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Feb 03 21:08:53 dist-test-slave-j889 krb5kdc[12896](info): set up 2 sockets
Feb 03 21:08:53 dist-test-slave-j889 krb5kdc[12896](info): commencing operation
krb5kdc: starting...
W20260203 21:08:55.672477 410 mini_kdc.cc:121] Time spent starting KDC: real 1.802s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/test-user.keytab.
Feb 03 21:08:55 dist-test-slave-j889 krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-02-03T21:08:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-02-03T21:08:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.102.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.102.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.102.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/rangeradmin_127.0.102.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.102.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/rangeradmin_127.0.102.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.102.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.102.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.102.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/rangerlookup_127.0.102.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.102.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/rangerlookup_127.0.102.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.102.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/HTTP_127.0.102.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.102.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/HTTP_127.0.102.148@KRBTEST.COM.keytab.
I20260203 21:08:55.811064 410 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3aGl8N/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/postgres -l logfile start
2026-02-03 21:08:58.003 UTC [12927] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-02-03 21:08:58.004 UTC [12927] LOG: listening on IPv4 address "127.0.102.148", port 37591
2026-02-03 21:08:58.006 UTC [12927] LOG: listening on Unix socket "/tmp/.s.PGSQL.37591"
2026-02-03 21:08:58.011 UTC [12932] LOG: database system was shut down at 2026-02-03 21:08:56 UTC
2026-02-03 21:08:58.016 UTC [12927] LOG: database system is ready to accept connections
I20260203 21:08:59.760325 410 mini_postgres.cc:96] Postgres bound to 37591
2026-02-03 21:08:59.765 UTC [12938] FATAL: database "slave" does not exist
127.0.102.148:37591 - accepting connections
I20260203 21:08:59.766242 410 mini_ranger.cc:161] Starting Ranger...
I20260203 21:08:59.782464 410 mini_ranger.cc:84] Created miniranger Postgres user
I20260203 21:08:59.843598 410 mini_ranger.cc:87] Created ranger Postgres database
I20260203 21:08:59.843683 410 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-admin
2026-02-03 21:09:00,113 [I] DB FLAVOR :POSTGRES
2026-02-03 21:09:00,114 [I] --------- Verifying Ranger DB connection ---------
2026-02-03 21:09:00,114 [I] Checking connection..
2026-02-03 21:09:00,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-02-03 21:09:00,434 [I] Checking connection passed.
2026-02-03 21:09:00,434 [I] --------- Verifying version history table ---------
2026-02-03 21:09:00,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-03 21:09:00,733 [I] Table x_db_version_h does not exist in database ranger
2026-02-03 21:09:00,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-03 21:09:01,046 [I] Table x_db_version_h does not exist in database ranger
2026-02-03 21:09:01,046 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-02-03 21:09:01,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-02-03 21:09:01.314 UTC [13005] WARNING: there is no transaction in progress
2026-02-03 21:09:01,320 [I] create_dbversion_catalog.sql file imported successfully
2026-02-03 21:09:01,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-02-03 21:09:01,609 [I] Table x_db_version_h already exists in database 'ranger'
2026-02-03 21:09:01,609 [I] --------- Importing Ranger Core DB Schema ---------
2026-02-03 21:09:01,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-02-03 21:09:01,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-02-03 21:09:02,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j889.c.gcp-upstream.internal','N') ;"
2026-02-03 21:09:02,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-02-03 21:09:02,787 [I] Table x_portal_user does not exist in database ranger
2026-02-03 21:09:02,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-02-03 21:09:03,114 [I] Table x_policy_ref_group does not exist in database ranger
2026-02-03 21:09:03,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-02-03 21:09:03,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-02-03 21:09:03,723 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-02-03 21:09:03,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-02-03 21:09:04.476 UTC [13122] WARNING: there is no transaction in progress
2026-02-03 21:09:04.490 UTC [13122] WARNING: there is no transaction in progress
2026-02-03 21:09:04.503 UTC [13122] WARNING: there is no transaction in progress
2026-02-03 21:09:04.516 UTC [13122] WARNING: there is no transaction in progress
2026-02-03 21:09:04.530 UTC [13122] WARNING: there is no transaction in progress
2026-02-03 21:09:04.543 UTC [13122] WARNING: there is no transaction in progress
2026-02-03 21:09:04.592 UTC [13122] WARNING: there is no transaction in progress
2026-02-03 21:09:04.596 UTC [13122] WARNING: there is no transaction in progress
2026-02-03 21:09:04.605 UTC [13122] WARNING: there is no transaction in progress
2026-02-03 21:09:04.614 UTC [13122] WARNING: there is no transaction in progress
2026-02-03 21:09:04.622 UTC [13122] WARNING: there is no transaction in progress
2026-02-03 21:09:04.632 UTC [13122] WARNING: there is no transaction in progress
2026-02-03 21:09:04.639 UTC [13122] WARNING: there is no transaction in progress
2026-02-03 21:09:04.643 UTC [13122] WARNING: there is no transaction in progress
2026-02-03 21:09:04.646 UTC [13122] WARNING: there is no transaction in progress
2026-02-03 21:09:04.649 UTC [13122] WARNING: there is no transaction in progress
2026-02-03 21:09:04,980 [I] ranger_core_db_postgres.sql file imported successfully
2026-02-03 21:09:04,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-02-03 21:09:05,257 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-02-03 21:09:05,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-02-03 21:09:05,545 [I] Table x_portal_user already exists in database 'ranger'
2026-02-03 21:09:05,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-02-03 21:09:05,842 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-02-03 21:09:05,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-02-03 21:09:06,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-02-03 21:09:06,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j889.c.gcp-upstream.internal';"
2026-02-03 21:09:06,720 [I] CORE_DB_SCHEMA import status has been updated
2026-02-03 21:09:06,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-02-03 21:09:07,014 [I] DB_PATCHES have already been applied
I20260203 21:09:07.020138 410 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-admin:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3aGl8N/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260203 21:09:07.020196 410 mini_ranger.cc:193] Using host: 127.0.102.148
I20260203 21:09:07.022723 410 mini_ranger.cc:227] Ranger admin URL: http://127.0.102.148:55943
Feb 03, 2026 9:09:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Feb 03, 2026 9:09:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Feb 03, 2026 9:09:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-admin/ews/webapp
Feb 03, 2026 9:09:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Feb 03, 2026 9:09:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-admin/ews/webapp] .....
Feb 03, 2026 9:09:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-admin/ews/webapp].
Feb 03, 2026 9:09:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260203 21:09:08.985739 410 mini_ranger.cc:160] Time spent starting Ranger: real 9.219s user 0.000s sys 0.008s
Feb 03, 2026 9:09:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Feb 03, 2026 9:09:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
Feb 03, 2026 9:09:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Feb 03, 2026 9:09:28 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Feb 03, 2026 9:09:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Feb 03, 2026 9:09:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Feb 03, 2026 9:09:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Feb 03, 2026 9:09:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Feb 03, 2026 9:09:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Feb 03, 2026 9:09:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Feb 03, 2026 9:09:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Feb 03, 2026 9:09:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Feb 03, 2026 9:09:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Feb 03, 2026 9:09:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Feb 03, 2026 9:09:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Feb 03, 2026 9:09:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Feb 03, 2026 9:09:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Feb 03, 2026 9:09:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260203 21:09:29.379150 410 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.102.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.102.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.102.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/rangerkms_127.0.102.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.102.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/rangerkms_127.0.102.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.102.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.102.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/HTTP_127.0.102.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.102.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/HTTP_127.0.102.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Feb 03 21:09:29 dist-test-slave-j889 krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152969, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260203 21:09:29.444532 410 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260203 21:09:29.453114 410 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260203 21:09:29.549391 410 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260203 21:09:29.549484 410 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-kms
I20260203 21:09:29.549501 410 mini_ranger_kms.cc:227] Using postgres at 127.0.102.148:37591
2026-02-03 21:09:29,701 [I] DB FLAVOR :POSTGRES
2026-02-03 21:09:29,702 [I] --------- Verifying Ranger DB connection ---------
2026-02-03 21:09:29,702 [I] Checking connection
2026-02-03 21:09:29,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-02-03 21:09:30,023 [I] connection success
2026-02-03 21:09:30,024 [I] --------- Verifying Ranger DB tables ---------
2026-02-03 21:09:30,024 [I] Verifying table ranger_masterkey in database rangerkms
2026-02-03 21:09:30,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-02-03 21:09:30,303 [I] Table ranger_masterkey does not exist in database rangerkms
2026-02-03 21:09:30,304 [I] --------- Importing Ranger Core DB Schema ---------
2026-02-03 21:09:30,304 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-02-03 21:09:30,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.102.148:37591/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-02-03 21:09:30,608 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260203 21:09:31.078552 410 mini_ranger_kms.cc:310] Created kms service
I20260203 21:09:31.308056 410 mini_ranger_kms.cc:326] Created kudu user
I20260203 21:09:31.346200 410 mini_ranger_kms.cc:343] Created rangerkms user
Feb 03 21:09:31 dist-test-slave-j889 krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152969, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.102.148@KRBTEST.COM
I20260203 21:09:31.734726 410 mini_ranger_kms.cc:384] Added ranger policy
I20260203 21:09:31.734961 410 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-kms:/tmp/dist-test-task3aGl8N/build/debug/bin/postgresql.jar:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3aGl8N/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260203 21:09:31.734980 410 mini_ranger_kms.cc:240] Using host: 127.0.102.148
I20260203 21:09:31.737896 410 mini_ranger_kms.cc:276] Ranger KMS PID: 13314
I20260203 21:09:31.737972 410 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.102.148:53687
Feb 03, 2026 9:09:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Feb 03, 2026 9:09:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Feb 03, 2026 9:09:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Feb 03, 2026 9:09:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-kms/ews/webapp] .....
Feb 03, 2026 9:09:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-kms/ews/webapp].
Feb 03, 2026 9:09:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260203 21:09:33.692181 410 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.248s user 0.013s sys 0.122s
I20260203 21:09:33.692298 410 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260203 21:09:33.692359 410 mini_ranger_kms.cc:401] 127.0.102.148:53687/kms/v1/keys
Feb 03 21:09:33 dist-test-slave-j889 krb5kdc[12896](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1770152973, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.102.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Feb 03 21:09:34 dist-test-slave-j889 krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152973, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.102.148@KRBTEST.COM for HTTP/127.0.102.148@KRBTEST.COM
Feb 03 21:09:34 dist-test-slave-j889 krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152973, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.102.148@KRBTEST.COM for HTTP/127.0.102.148@KRBTEST.COM
Feb 03 21:09:36 dist-test-slave-j889 krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:09:36.872776 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:34597
--webserver_interface=127.0.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:43221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.102.190:34597
--ranger_config_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.102.148:53687/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:09:36.973227 13367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:09:36.973469 13367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:09:36.973531 13367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:09:36.976881 13367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260203 21:09:36.976943 13367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:09:36.976976 13367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:09:36.977008 13367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260203 21:09:36.977038 13367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260203 21:09:36.981387 13367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:43221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.102.148:53687/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.102.190:34597
--ranger_config_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.102.190:34597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.102.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:09:36.982385 13367 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:09:36.983211 13367 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:09:36.988809 13375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:09:36.988857 13373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:09:36.988814 13372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:09:36.988896 13367 server_base.cc:1060] running on GCE node
I20260203 21:09:36.989517 13367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:09:36.990366 13367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:09:36.991506 13367 hybrid_clock.cc:648] HybridClock initialized: now 1770152976991483 us; error 35 us; skew 500 ppm
Feb 03 21:09:36 dist-test-slave-j889 krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:09:36.993831 13367 init.cc:377] Logged in from keytab as kudu/127.0.102.190@KRBTEST.COM (short username kudu)
I20260203 21:09:36.994776 13367 webserver.cc:492] Webserver started at http://127.0.102.190:35131/ using document root <none> and password file <none>
I20260203 21:09:36.995254 13367 fs_manager.cc:362] Metadata directory not provided
I20260203 21:09:36.995302 13367 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:09:36.995465 13367 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 03 21:09:36 dist-test-slave-j889 krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.190@KRBTEST.COM for HTTP/127.0.102.148@KRBTEST.COM
I20260203 21:09:37.040443 13367 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/data/instance:
uuid: "76d4d3cf6140425aa18bde46d21b57aa"
format_stamp: "Formatted at 2026-02-03 21:09:37 on dist-test-slave-j889"
server_key: "b1edbd5f10dc25ed6d05ada5f1c482d3"
server_key_iv: "8ee3f22a688e8a13f485a8ef7811e44b"
server_key_version: "kuduclusterkey@0"
I20260203 21:09:37.041003 13367 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/wal/instance:
uuid: "76d4d3cf6140425aa18bde46d21b57aa"
format_stamp: "Formatted at 2026-02-03 21:09:37 on dist-test-slave-j889"
server_key: "b1edbd5f10dc25ed6d05ada5f1c482d3"
server_key_iv: "8ee3f22a688e8a13f485a8ef7811e44b"
server_key_version: "kuduclusterkey@0"
I20260203 21:09:37.044402 13367 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260203 21:09:37.057899 13382 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:09:37.058990 13367 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260203 21:09:37.059118 13367 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/wal
uuid: "76d4d3cf6140425aa18bde46d21b57aa"
format_stamp: "Formatted at 2026-02-03 21:09:37 on dist-test-slave-j889"
server_key: "b1edbd5f10dc25ed6d05ada5f1c482d3"
server_key_iv: "8ee3f22a688e8a13f485a8ef7811e44b"
server_key_version: "kuduclusterkey@0"
I20260203 21:09:37.059224 13367 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:09:37.088439 13367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:09:37.090802 13367 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:09:37.090972 13367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:09:37.098217 13434 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.190:34597 every 8 connection(s)
I20260203 21:09:37.098238 13367 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.190:34597
I20260203 21:09:37.099268 13367 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/data/info.pb
I20260203 21:09:37.101969 13435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260203 21:09:37.106768 13435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa: Bootstrap starting.
I20260203 21:09:37.108482 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 13367
I20260203 21:09:37.108588 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/wal/instance
I20260203 21:09:37.108837 410 external_mini_cluster.cc:1468] Setting key 9bc797753af60fc7472f878fdbeea8f9
I20260203 21:09:37.109001 13435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa: Neither blocks nor log segments found. Creating new log.
I20260203 21:09:37.109831 13435 log.cc:826] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa: Log is configured to *not* fsync() on all Append() calls
I20260203 21:09:37.111778 13435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa: No bootstrap required, opened a new log
I20260203 21:09:37.115072 13435 raft_consensus.cc:359] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d4d3cf6140425aa18bde46d21b57aa" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 34597 } }
I20260203 21:09:37.115247 13435 raft_consensus.cc:385] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260203 21:09:37.115290 13435 raft_consensus.cc:740] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d4d3cf6140425aa18bde46d21b57aa, State: Initialized, Role: FOLLOWER
I20260203 21:09:37.115726 13435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d4d3cf6140425aa18bde46d21b57aa" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 34597 } }
Feb 03 21:09:37 dist-test-slave-j889 krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:09:37.115840 13435 raft_consensus.cc:399] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260203 21:09:37.115898 13435 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260203 21:09:37.115981 13435 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [term 0 FOLLOWER]: Advancing to term 1
I20260203 21:09:37.117007 13435 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d4d3cf6140425aa18bde46d21b57aa" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 34597 } }
I20260203 21:09:37.117393 13435 leader_election.cc:304] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76d4d3cf6140425aa18bde46d21b57aa; no voters:
I20260203 21:09:37.117705 13435 leader_election.cc:290] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260203 21:09:37.117797 13440 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [term 1 FOLLOWER]: Leader election won for term 1
I20260203 21:09:37.117995 13440 raft_consensus.cc:697] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [term 1 LEADER]: Becoming Leader. State: Replica: 76d4d3cf6140425aa18bde46d21b57aa, State: Running, Role: LEADER
I20260203 21:09:37.118629 13440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d4d3cf6140425aa18bde46d21b57aa" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 34597 } }
I20260203 21:09:37.118829 13435 sys_catalog.cc:565] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [sys.catalog]: configured and running, proceeding with master startup.
I20260203 21:09:37.120210 13441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76d4d3cf6140425aa18bde46d21b57aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d4d3cf6140425aa18bde46d21b57aa" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 34597 } } }
I20260203 21:09:37.120286 13442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76d4d3cf6140425aa18bde46d21b57aa. Latest consensus state: current_term: 1 leader_uuid: "76d4d3cf6140425aa18bde46d21b57aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d4d3cf6140425aa18bde46d21b57aa" member_type: VOTER last_known_addr { host: "127.0.102.190" port: 34597 } } }
I20260203 21:09:37.120389 13441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [sys.catalog]: This master's current role is: LEADER
I20260203 21:09:37.120404 13442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa [sys.catalog]: This master's current role is: LEADER
I20260203 21:09:37.120744 13452 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260203 21:09:37.121037 13438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:09:37.109808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.0.102.190:34597)
0203 21:09:37.110213 (+ 405us) server_negotiation.cc:207] Beginning negotiation
0203 21:09:37.110225 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0203 21:09:37.110260 (+ 35us) server_negotiation.cc:408] Connection header received
0203 21:09:37.110868 (+ 608us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:09:37.110890 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:09:37.111224 (+ 334us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:09:37.111555 (+ 331us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:09:37.112354 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:09:37.113505 (+ 1151us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:09:37.114116 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:09:37.114414 (+ 298us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:09:37.116359 (+ 1945us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:09:37.116381 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:09:37.116395 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:09:37.116434 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:09:37.118214 (+ 1780us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:09:37.118581 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:09:37.118586 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:09:37.118591 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:09:37.118677 (+ 86us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:09:37.118904 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:09:37.118907 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:09:37.118909 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:09:37.119328 (+ 419us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:09:37.119531 (+ 203us) server_negotiation.cc:1036] Waiting for connection context
0203 21:09:37.119809 (+ 278us) server_negotiation.cc:300] Negotiation successful
0203 21:09:37.120040 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":125,"threads_started":1}
I20260203 21:09:37.121037 13435 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260203 21:09:37.123772 13452 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260203 21:09:37.132712 13452 catalog_manager.cc:1357] Generated new cluster ID: edc7f876116b4ee9acb3975da84e26e9
I20260203 21:09:37.132791 13452 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260203 21:09:37.145833 13452 catalog_manager.cc:1380] Generated new certificate authority record
I20260203 21:09:37.147294 13452 catalog_manager.cc:1514] Loading token signing keys...
I20260203 21:09:37.156945 13452 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 76d4d3cf6140425aa18bde46d21b57aa: Generated new TSK 0
I20260203 21:09:37.157846 13452 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab.
I20260203 21:09:37.836534 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.102.148:53687/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.129:0
--local_ip_for_outbound_sockets=127.0.102.129
--webserver_interface=127.0.102.129
--webserver_port=0
--tserver_master_addrs=127.0.102.190:34597
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:43221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260203 21:09:37.968210 13481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:09:37.968426 13481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:09:37.968510 13481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:09:37.971801 13481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:09:37.971866 13481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:09:37.971966 13481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.129
I20260203 21:09:37.976341 13481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:43221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.102.148:53687/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.102.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:34597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.129
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:09:37.977448 13481 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:09:37.978281 13481 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:09:37.984540 13489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:09:37.984540 13488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:09:37.984567 13491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:09:37.985059 13481 server_base.cc:1060] running on GCE node
I20260203 21:09:37.985390 13481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:09:37.985903 13481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:09:37.987056 13481 hybrid_clock.cc:648] HybridClock initialized: now 1770152977987039 us; error 36 us; skew 500 ppm
Feb 03 21:09:37 dist-test-slave-j889 krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:09:37.989408 13481 init.cc:377] Logged in from keytab as kudu/127.0.102.129@KRBTEST.COM (short username kudu)
I20260203 21:09:37.990399 13481 webserver.cc:492] Webserver started at http://127.0.102.129:43841/ using document root <none> and password file <none>
I20260203 21:09:37.990931 13481 fs_manager.cc:362] Metadata directory not provided
I20260203 21:09:37.990978 13481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:09:37.991142 13481 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 03 21:09:37 dist-test-slave-j889 krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for HTTP/127.0.102.148@KRBTEST.COM
I20260203 21:09:38.006294 13481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/data/instance:
uuid: "1c60dbf227de4f7c992b88c867afa980"
format_stamp: "Formatted at 2026-02-03 21:09:37 on dist-test-slave-j889"
server_key: "5e1c9fcc8b02dc6cf786ad3f3fd596b0"
server_key_iv: "883f876fac78f6d183de1fac847bef4f"
server_key_version: "kuduclusterkey@0"
I20260203 21:09:38.006916 13481 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/wal/instance:
uuid: "1c60dbf227de4f7c992b88c867afa980"
format_stamp: "Formatted at 2026-02-03 21:09:37 on dist-test-slave-j889"
server_key: "5e1c9fcc8b02dc6cf786ad3f3fd596b0"
server_key_iv: "883f876fac78f6d183de1fac847bef4f"
server_key_version: "kuduclusterkey@0"
I20260203 21:09:38.013969 13481 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260203 21:09:38.031808 13498 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:09:38.033254 13481 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260203 21:09:38.033442 13481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/wal
uuid: "1c60dbf227de4f7c992b88c867afa980"
format_stamp: "Formatted at 2026-02-03 21:09:37 on dist-test-slave-j889"
server_key: "5e1c9fcc8b02dc6cf786ad3f3fd596b0"
server_key_iv: "883f876fac78f6d183de1fac847bef4f"
server_key_version: "kuduclusterkey@0"
I20260203 21:09:38.033638 13481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:09:38.055217 13481 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:09:38.058660 13481 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:09:38.058915 13481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:09:38.059607 13481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:09:38.060765 13481 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:09:38.060869 13481 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:09:38.060987 13481 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:09:38.061069 13481 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:09:38.071859 13481 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.129:43953
I20260203 21:09:38.072172 13611 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.129:43953 every 8 connection(s)
I20260203 21:09:38.073046 13481 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/data/info.pb
I20260203 21:09:38.082247 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 13481
I20260203 21:09:38.082373 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-0/wal/instance
I20260203 21:09:38.082592 410 external_mini_cluster.cc:1468] Setting key 7436b5e6a128f646ddac871515ffbc9a
Feb 03 21:09:38 dist-test-slave-j889 krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.129@KRBTEST.COM for kudu/127.0.102.190@KRBTEST.COM
I20260203 21:09:38.088192 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:09:38.075302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.102.129:40809 (local address 127.0.102.190:34597)
0203 21:09:38.078079 (+ 2777us) server_negotiation.cc:207] Beginning negotiation
0203 21:09:38.078083 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0203 21:09:38.078420 (+ 337us) server_negotiation.cc:408] Connection header received
0203 21:09:38.079256 (+ 836us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0203 21:09:38.079260 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0203 21:09:38.079317 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0203 21:09:38.079398 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0203 21:09:38.080533 (+ 1135us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:09:38.081166 (+ 633us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0203 21:09:38.081768 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0203 21:09:38.081946 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:09:38.084102 (+ 2156us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0203 21:09:38.084121 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0203 21:09:38.084124 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0203 21:09:38.084147 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0203 21:09:38.085831 (+ 1684us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:09:38.086358 (+ 527us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:09:38.086362 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:09:38.086364 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:09:38.086420 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0203 21:09:38.086747 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0203 21:09:38.086751 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0203 21:09:38.086753 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0203 21:09:38.086938 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0203 21:09:38.087011 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0203 21:09:38.087709 (+ 698us) server_negotiation.cc:300] Negotiation successful
0203 21:09:38.087840 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2686,"thread_start_us":115,"threads_started":1}
I20260203 21:09:38.089701 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:09:38.075747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34597 (local address 127.0.102.129:40809)
0203 21:09:38.078284 (+ 2537us) negotiation.cc:107] Waiting for socket to connect
0203 21:09:38.078317 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0203 21:09:38.079069 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:09:38.079568 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0203 21:09:38.079576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0203 21:09:38.079843 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0203 21:09:38.080386 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:09:38.080398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:09:38.081282 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0203 21:09:38.081284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0203 21:09:38.081664 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0203 21:09:38.081671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0203 21:09:38.081882 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0203 21:09:38.082465 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0203 21:09:38.082498 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0203 21:09:38.083950 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0203 21:09:38.085945 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:09:38.085951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:09:38.085962 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:09:38.086253 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:09:38.086521 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0203 21:09:38.086524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0203 21:09:38.086525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0203 21:09:38.086635 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0203 21:09:38.087097 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0203 21:09:38.087103 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0203 21:09:38.087396 (+ 293us) client_negotiation.cc:769] Sending connection context
0203 21:09:38.087640 (+ 244us) client_negotiation.cc:240] Negotiation successful
0203 21:09:38.087896 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2373,"thread_start_us":111,"threads_started":1}
I20260203 21:09:38.091001 13612 heartbeater.cc:344] Connected to a master server at 127.0.102.190:34597
I20260203 21:09:38.091243 13612 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.0.102.130@KRBTEST.COM; defaulting to no policy
I20260203 21:09:38.091837 13612 heartbeater.cc:507] Master 127.0.102.190:34597 requested a full tablet report, sending...
I20260203 21:09:38.093623 13399 ts_manager.cc:194] Registered new tserver with Master: 1c60dbf227de4f7c992b88c867afa980 (127.0.102.129:43953)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.102.130@KRBTEST.COM" created.
I20260203 21:09:38.095436 13399 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.102.129@KRBTEST.COM'} at 127.0.102.129:40809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.102.130@KRBTEST.COM" created.
Exception in thread "main" Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab.
org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.0.102.190 from keytab /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
I20260203 21:09:38.143600 410 external_mini_cluster.cc:1366] Running /tmp/dist-test-task3aGl8N/build/debug/bin/kudu
/tmp/dist-test-task3aGl8N/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.102.148:53687/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.102.130:0
--local_ip_for_outbound_sockets=127.0.102.130
--webserver_interface=127.0.102.130
--webserver_port=0
--tserver_master_addrs=127.0.102.190:34597
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.102.148:43221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
I20260203 21:09:38.161226 13473 server.cc:273] Received an EOF from the subprocess
F20260203 21:09:38.162070 13459 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7fd548bc3dcd google::LogMessage::Fail() at ??:0
@ 0x7fd548bc7b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fd548bc37cc google::LogMessage::Flush() at ??:0
@ 0x7fd548bc4f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd54a87b2d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7fd54a8780d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7fd54a87c005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x560af04daa0a std::function<>::operator()() at ??:0
@ 0x7fd54a01db04 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd54a8b76db start_thread at ??:0
@ 0x7fd547d7a71f clone at ??:0
W20260203 21:09:38.276659 13623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260203 21:09:38.276961 13623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260203 21:09:38.277068 13623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260203 21:09:38.281625 13623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260203 21:09:38.281719 13623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260203 21:09:38.281862 13623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.102.130
I20260203 21:09:38.286914 13623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.102.148:43221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.102.148:53687/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.102.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.102.190:34597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.102.130
--log_dir=/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83
build type DEBUG
built by None at 03 Feb 2026 20:50:43 UTC on 24a791456cd2
build id 10330
I20260203 21:09:38.288129 13623 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260203 21:09:38.289155 13623 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260203 21:09:38.295831 13639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:09:38.295926 13642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260203 21:09:38.295831 13640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260203 21:09:38.296293 13623 server_base.cc:1060] running on GCE node
I20260203 21:09:38.296708 13623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260203 21:09:38.297271 13623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260203 21:09:38.298432 13623 hybrid_clock.cc:648] HybridClock initialized: now 1770152978298406 us; error 44 us; skew 500 ppm
Feb 03 21:09:38 dist-test-slave-j889 krb5kdc[12896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260203 21:09:38.301008 13623 init.cc:377] Logged in from keytab as kudu/127.0.102.130@KRBTEST.COM (short username kudu)
I20260203 21:09:38.302022 13623 webserver.cc:492] Webserver started at http://127.0.102.130:37345/ using document root <none> and password file <none>
I20260203 21:09:38.302551 13623 fs_manager.cc:362] Metadata directory not provided
I20260203 21:09:38.302600 13623 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260203 21:09:38.302770 13623 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Feb 03 21:09:38 dist-test-slave-j889 krb5kdc[12896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1770152978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.102.130@KRBTEST.COM for HTTP/127.0.102.148@KRBTEST.COM
I20260203 21:09:38.315150 13623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/data/instance:
uuid: "7f4d563b9ccc4f1196c4c1a51526f914"
format_stamp: "Formatted at 2026-02-03 21:09:38 on dist-test-slave-j889"
server_key: "2d675146c582ddb8c9d4cd3ad1c71b59"
server_key_iv: "dfa383aa37494ec28c7f50de05dc127f"
server_key_version: "kuduclusterkey@0"
I20260203 21:09:38.315717 13623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/wal/instance:
uuid: "7f4d563b9ccc4f1196c4c1a51526f914"
format_stamp: "Formatted at 2026-02-03 21:09:38 on dist-test-slave-j889"
server_key: "2d675146c582ddb8c9d4cd3ad1c71b59"
server_key_iv: "dfa383aa37494ec28c7f50de05dc127f"
server_key_version: "kuduclusterkey@0"
I20260203 21:09:38.319387 13623 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260203 21:09:38.332984 13649 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260203 21:09:38.334385 13623 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260203 21:09:38.334527 13623 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/data,/tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/wal
uuid: "7f4d563b9ccc4f1196c4c1a51526f914"
format_stamp: "Formatted at 2026-02-03 21:09:38 on dist-test-slave-j889"
server_key: "2d675146c582ddb8c9d4cd3ad1c71b59"
server_key_iv: "dfa383aa37494ec28c7f50de05dc127f"
server_key_version: "kuduclusterkey@0"
I20260203 21:09:38.334652 13623 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260203 21:09:38.358979 13623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260203 21:09:38.362313 13623 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260203 21:09:38.362509 13623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260203 21:09:38.363202 13623 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260203 21:09:38.364259 13623 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260203 21:09:38.364320 13623 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260203 21:09:38.364372 13623 ts_tablet_manager.cc:616] Registered 0 tablets
I20260203 21:09:38.364398 13623 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260203 21:09:38.375268 13623 rpc_server.cc:307] RPC server started. Bound to: 127.0.102.130:43737
I20260203 21:09:38.375646 13762 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.102.130:43737 every 8 connection(s)
I20260203 21:09:38.376551 13623 server_base.cc:1192] Dumped server information to /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/data/info.pb
I20260203 21:09:38.379695 410 external_mini_cluster.cc:1428] Started /tmp/dist-test-task3aGl8N/build/debug/bin/kudu as pid 13623
I20260203 21:09:38.379832 410 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task3aGl8N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1770152788143875-410-0/minicluster-data/ts-1/wal/instance
I20260203 21:09:38.380087 410 external_mini_cluster.cc:1468] Setting key 074d7b6cefa8f792e3fee710fbed3173
W20260203 21:09:38.397475 12894 connection.cc:537] client connection to 127.0.102.190:34597 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260203 21:09:38.398449 13765 negotiation.cc:337] Failed RPC negotiation. Trace:
0203 21:09:38.378995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34597 (local address 127.0.102.130:56229)
0203 21:09:38.380735 (+ 1740us) negotiation.cc:107] Waiting for socket to connect
0203 21:09:38.380783 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0203 21:09:38.381745 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0203 21:09:38.397804 (+ 16059us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:34597: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":118,"threads_started":1}
W20260203 21:09:38.398967 13763 heartbeater.cc:646] Failed to heartbeat to 127.0.102.190:34597 (0 consecutive failures): Network error: Failed to ping master at 127.0.102.190:34597: Client connection negotiation failed: client connection to 127.0.102.190:34597: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260203 21:09:38.399632 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:09:38.399406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34597 (local address 127.0.102.130:40393)
0203 21:09:38.399526 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0203 21:09:38.399578 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260203 21:09:38.400238 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:09:38.400066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34597 (local address 127.0.102.130:37901)
0203 21:09:38.400155 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0203 21:09:38.400191 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260203 21:09:38.400377 13763 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260203 21:09:39.099961 13612 heartbeater.cc:499] Master 127.0.102.190:34597 was elected leader, sending a full tablet report...
I20260203 21:09:39.100874 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:09:39.100417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34597 (local address 127.0.102.129:45633)
0203 21:09:39.100703 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0203 21:09:39.100770 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
W20260203 21:09:39.101133 13612 heartbeater.cc:646] Failed to heartbeat to 127.0.102.190:34597 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.102.190:34597: connect: Connection refused (error 111)
I20260203 21:09:39.401422 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:09:39.400989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34597 (local address 127.0.102.130:37825)
0203 21:09:39.401253 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0203 21:09:39.401310 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
I20260203 21:09:40.102165 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:09:40.101733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34597 (local address 127.0.102.129:51949)
0203 21:09:40.102001 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0203 21:09:40.102068 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":87,"threads_started":1}
I20260203 21:09:40.402556 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:09:40.402139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34597 (local address 127.0.102.130:54745)
0203 21:09:40.402389 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0203 21:09:40.402445 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":86,"threads_started":1}
I20260203 21:09:41.103307 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:09:41.102905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34597 (local address 127.0.102.129:52039)
0203 21:09:41.103153 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0203 21:09:41.103212 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":77,"threads_started":1}
I20260203 21:09:41.403719 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:09:41.403288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34597 (local address 127.0.102.130:53881)
0203 21:09:41.403549 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0203 21:09:41.403606 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
I20260203 21:09:42.104602 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:09:42.104172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34597 (local address 127.0.102.129:49645)
0203 21:09:42.104447 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0203 21:09:42.104504 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
I20260203 21:09:42.404912 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:09:42.404464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34597 (local address 127.0.102.130:60199)
0203 21:09:42.404757 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0203 21:09:42.404815 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":95,"threads_started":1}
I20260203 21:09:43.105707 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:09:43.105317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34597 (local address 127.0.102.129:49449)
0203 21:09:43.105559 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0203 21:09:43.105614 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260203 21:09:43.406075 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0203 21:09:43.405644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.102.190:34597 (local address 127.0.102.130:48277)
0203 21:09:43.405926 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0203 21:09:43.405981 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.102.190:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":92,"threads_started":1}
F20260203 21:09:43.424199 410 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1770152983 (unix time) try "date -d @1770152983" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000019a) received by PID 410 (TID 0x7fd60c404d80) from PID 410; stack trace: ***
@ 0x7fd6180bf980 (unknown) at ??:0
@ 0x7fd6156f5fb7 gsignal at ??:0
@ 0x7fd6156f7921 abort at ??:0
@ 0x7fd616963dcd google::LogMessage::Fail() at ??:0
@ 0x7fd616967b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fd6169637cc google::LogMessage::Flush() at ??:0
@ 0x7fd616964f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd6196f0cab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7fd6196ed9c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x55bd45a2392f kudu::SecurityITest::StartCluster() at ??:0
@ 0x55bd45a0bc15 kudu::SecurityITest_TestEncryptionWithKMSIntegrationMultipleServers_Test::TestBody() at ??:0
@ 0x7fd6179740ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fd617968bdb testing::Test::Run() at ??:0
@ 0x7fd617968d9d testing::TestInfo::Run() at ??:0
@ 0x7fd617969377 testing::TestSuite::Run() at ??:0
@ 0x7fd61796977c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7fd61797460d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fd617968e63 testing::UnitTest::Run() at ??:0
@ 0x55bd45a5bc7e RUN_ALL_TESTS() at ??:0
@ 0x55bd45a5ae6b main at ??:0
@ 0x7fd6156d8bf7 __libc_start_main at ??:0
@ 0x55bd459f306a _start at ??:0
2026-02-03 21:09:43.591 UTC [13242] FATAL: terminating connection due to unexpected postmaster exit
2026-02-03 21:09:43.591 UTC [13345] FATAL: terminating connection due to unexpected postmaster exit
2026-02-03 21:09:43.592 UTC [13237] FATAL: terminating connection due to unexpected postmaster exit
2026-02-03 21:09:43.593 UTC [13241] FATAL: terminating connection due to unexpected postmaster exit
2026-02-03 21:09:43.593 UTC [13239] FATAL: terminating connection due to unexpected postmaster exit
2026-02-03 21:09:43.593 UTC [13238] FATAL: terminating connection due to unexpected postmaster exit