[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:01:59 dist-test-slave-m28t krb5kdc[13673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:01:59 dist-test-slave-m28t krb5kdc[13673](info): set up 2 sockets
Jan 28 15:01:59 dist-test-slave-m28t krb5kdc[13673](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260128 15:02:00.888168 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.821s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Jan 28 15:02:00 dist-test-slave-m28t krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:02:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:01.013113 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:41231
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:41231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:01.117918 13689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:01.118163 13689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:01.118237 13689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:01.121554 13689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:01.121620 13689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:01.121652 13689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:01.121677 13689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:01.121711 13689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:02:01.125957 13689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:41231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:41231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:01.126997 13689 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:01.127822 13689 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:01.133437 13694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:01.133533 13689 server_base.cc:1060] running on GCE node
W20260128 15:02:01.133626 13695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:01.133533 13697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:01.134176 13689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:01.135017 13689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:01.136164 13689 hybrid_clock.cc:648] HybridClock initialized: now 1769612521136122 us; error 59 us; skew 500 ppm
Jan 28 15:02:01 dist-test-slave-m28t krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612521, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:01.138584 13689 init.cc:377] Logged in from keytab as kudu/127.13.88.190@KRBTEST.COM (short username kudu)
I20260128 15:02:01.139633 13689 webserver.cc:492] Webserver started at http://127.13.88.190:37825/ using document root <none> and password file <none>
I20260128 15:02:01.140141 13689 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:01.140189 13689 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:01.140368 13689 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:01.142062 13689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "246f4cc35b8b40a0ab2c6f7a280ee46e"
format_stamp: "Formatted at 2026-01-28 15:02:01 on dist-test-slave-m28t"
server_key: "de17b50eb515e469fcd27b5b3cdfa4eb"
server_key_iv: "63ef5b93c7d633b5e06ff0969492b8c0"
server_key_version: "encryptionkey@0"
I20260128 15:02:01.142531 13689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "246f4cc35b8b40a0ab2c6f7a280ee46e"
format_stamp: "Formatted at 2026-01-28 15:02:01 on dist-test-slave-m28t"
server_key: "de17b50eb515e469fcd27b5b3cdfa4eb"
server_key_iv: "63ef5b93c7d633b5e06ff0969492b8c0"
server_key_version: "encryptionkey@0"
I20260128 15:02:01.145673 13689 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260128 15:02:01.147684 13704 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:01.148658 13689 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260128 15:02:01.148763 13689 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "246f4cc35b8b40a0ab2c6f7a280ee46e"
format_stamp: "Formatted at 2026-01-28 15:02:01 on dist-test-slave-m28t"
server_key: "de17b50eb515e469fcd27b5b3cdfa4eb"
server_key_iv: "63ef5b93c7d633b5e06ff0969492b8c0"
server_key_version: "encryptionkey@0"
I20260128 15:02:01.148842 13689 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:01.166253 13689 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:01.169320 13689 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:01.169502 13689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:01.176234 13756 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:41231 every 8 connection(s)
I20260128 15:02:01.176241 13689 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:41231
I20260128 15:02:01.177245 13689 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:01.178524 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 13689
I20260128 15:02:01.178670 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:01.179877 13666 external_mini_cluster.cc:1468] Setting key f43d9f249f3fce43d6f8517116f58ec1
I20260128 15:02:01.180397 13757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:01.186592 13757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e: Bootstrap starting.
I20260128 15:02:01.189007 13757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:01.189752 13757 log.cc:826] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:01.191723 13757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e: No bootstrap required, opened a new log
Jan 28 15:02:01 dist-test-slave-m28t krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:01.194550 13757 raft_consensus.cc:359] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246f4cc35b8b40a0ab2c6f7a280ee46e" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41231 } }
I20260128 15:02:01.194725 13757 raft_consensus.cc:385] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:01.194797 13757 raft_consensus.cc:740] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 246f4cc35b8b40a0ab2c6f7a280ee46e, State: Initialized, Role: FOLLOWER
I20260128 15:02:01.195169 13757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246f4cc35b8b40a0ab2c6f7a280ee46e" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41231 } }
I20260128 15:02:01.195295 13757 raft_consensus.cc:399] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:01.195358 13757 raft_consensus.cc:493] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:01.195438 13757 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:01.196319 13757 raft_consensus.cc:515] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246f4cc35b8b40a0ab2c6f7a280ee46e" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41231 } }
I20260128 15:02:01.196697 13757 leader_election.cc:304] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 246f4cc35b8b40a0ab2c6f7a280ee46e; no voters:
I20260128 15:02:01.196957 13757 leader_election.cc:290] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:01.197124 13762 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:01.197366 13762 raft_consensus.cc:697] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [term 1 LEADER]: Becoming Leader. State: Replica: 246f4cc35b8b40a0ab2c6f7a280ee46e, State: Running, Role: LEADER
I20260128 15:02:01.197696 13762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246f4cc35b8b40a0ab2c6f7a280ee46e" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41231 } }
I20260128 15:02:01.197997 13757 sys_catalog.cc:565] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:01.198298 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:01.182409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.13.88.190:41231)
0128 15:02:01.183168 (+ 759us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:01.183177 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:01.183236 (+ 59us) server_negotiation.cc:408] Connection header received
0128 15:02:01.185871 (+ 2635us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:01.185885 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:01.186152 (+ 267us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:01.186422 (+ 270us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:01.188476 (+ 2054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.189225 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:01.190039 (+ 814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.190270 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:01.193576 (+ 3306us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:01.193600 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:01.193609 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:01.193640 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:01.195239 (+ 1599us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:01.195733 (+ 494us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:01.195739 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:01.195744 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:01.195831 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:01.196045 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:01.196049 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:01.196051 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:01.196486 (+ 435us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:01.196689 (+ 203us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:01.197143 (+ 454us) server_negotiation.cc:300] Negotiation successful
0128 15:02:01.197378 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":227,"threads_started":1}
I20260128 15:02:01.199551 13762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 246f4cc35b8b40a0ab2c6f7a280ee46e. Latest consensus state: current_term: 1 leader_uuid: "246f4cc35b8b40a0ab2c6f7a280ee46e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246f4cc35b8b40a0ab2c6f7a280ee46e" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41231 } } }
I20260128 15:02:01.199645 13762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:01.200178 13776 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:01.200505 13763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "246f4cc35b8b40a0ab2c6f7a280ee46e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "246f4cc35b8b40a0ab2c6f7a280ee46e" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41231 } } }
I20260128 15:02:01.200579 13763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:01.203044 13776 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:01.208151 13776 catalog_manager.cc:1357] Generated new cluster ID: c251f5f69d9f40b08d7e116676cc543b
I20260128 15:02:01.208205 13776 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:01.218240 13776 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:01.219362 13776 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:01.232932 13776 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 246f4cc35b8b40a0ab2c6f7a280ee46e: Generated new TSK 0
I20260128 15:02:01.233582 13776 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:01.280493 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:41231
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:01.388855 13785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:01.389062 13785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:01.389132 13785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:01.392398 13785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:01.392514 13785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:01.392622 13785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:02:01.396845 13785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:41231
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:01.397876 13785 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:01.398679 13785 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:01.404803 13790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:01.404862 13791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:01.404898 13793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:01.405025 13785 server_base.cc:1060] running on GCE node
I20260128 15:02:01.405346 13785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:01.405813 13785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:01.406967 13785 hybrid_clock.cc:648] HybridClock initialized: now 1769612521406953 us; error 25 us; skew 500 ppm
Jan 28 15:02:01 dist-test-slave-m28t krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612521, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:01.409402 13785 init.cc:377] Logged in from keytab as kudu/127.13.88.129@KRBTEST.COM (short username kudu)
I20260128 15:02:01.410377 13785 webserver.cc:492] Webserver started at http://127.13.88.129:33219/ using document root <none> and password file <none>
I20260128 15:02:01.410892 13785 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:01.410938 13785 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:01.411118 13785 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:01.412813 13785 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "14a05d66fcc248a5b0d4d4c953778345"
format_stamp: "Formatted at 2026-01-28 15:02:01 on dist-test-slave-m28t"
server_key: "5ea82fe585f886986613684b551dd6e4"
server_key_iv: "6d0ee15e86b97dcd5bc61b4e5ed40e78"
server_key_version: "encryptionkey@0"
I20260128 15:02:01.413250 13785 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "14a05d66fcc248a5b0d4d4c953778345"
format_stamp: "Formatted at 2026-01-28 15:02:01 on dist-test-slave-m28t"
server_key: "5ea82fe585f886986613684b551dd6e4"
server_key_iv: "6d0ee15e86b97dcd5bc61b4e5ed40e78"
server_key_version: "encryptionkey@0"
I20260128 15:02:01.416438 13785 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260128 15:02:01.418478 13800 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:01.419438 13785 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:01.419536 13785 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "14a05d66fcc248a5b0d4d4c953778345"
format_stamp: "Formatted at 2026-01-28 15:02:01 on dist-test-slave-m28t"
server_key: "5ea82fe585f886986613684b551dd6e4"
server_key_iv: "6d0ee15e86b97dcd5bc61b4e5ed40e78"
server_key_version: "encryptionkey@0"
I20260128 15:02:01.419628 13785 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:01.428663 13785 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:01.431617 13785 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:01.431792 13785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:01.432376 13785 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:01.433387 13785 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:01.433444 13785 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:01.433498 13785 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:01.433542 13785 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:01.443239 13785 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:36311
I20260128 15:02:01.444506 13785 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:02:01.445374 13913 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:36311 every 8 connection(s)
I20260128 15:02:01.450781 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 13785
I20260128 15:02:01.450865 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:02:01.451077 13666 external_mini_cluster.cc:1468] Setting key 748205cfafd2acb24c3942617f37fcce
Jan 28 15:02:01 dist-test-slave-m28t krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612521, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:01.457664 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:01.447332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:54011 (local address 127.13.88.190:41231)
0128 15:02:01.447483 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:01.447488 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:01.448114 (+ 626us) server_negotiation.cc:408] Connection header received
0128 15:02:01.448991 (+ 877us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:01.448995 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:01.449052 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:01.449125 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:01.450742 (+ 1617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.451325 (+ 583us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:01.452018 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.452198 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:01.454386 (+ 2188us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:01.454404 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:01.454406 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:01.454430 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:01.455714 (+ 1284us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:01.456246 (+ 532us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:01.456248 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:01.456250 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:01.456286 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:01.456623 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:01.456625 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:01.456627 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:01.456766 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:01.456836 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:01.457431 (+ 595us) server_negotiation.cc:300] Negotiation successful
0128 15:02:01.457534 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260128 15:02:01.458492 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:01.447565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41231 (local address 127.13.88.129:54011)
0128 15:02:01.447976 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:01.448013 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:01.448797 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:01.449294 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:01.449302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:01.449690 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:01.450594 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:01.450609 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.451449 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:01.451453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:01.451910 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:01.451918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.452124 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:01.452725 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:01.452752 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:01.454247 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:01.455823 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:01.455828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:01.455840 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:01.456158 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:01.456365 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:01.456369 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:01.456371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:01.456521 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:01.456860 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:01.456867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:01.457163 (+ 296us) client_negotiation.cc:769] Sending connection context
0128 15:02:01.457405 (+ 242us) client_negotiation.cc:240] Negotiation successful
0128 15:02:01.457681 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.13.88.130@KRBTEST.COM; defaulting to no policy
I20260128 15:02:01.459709 13914 heartbeater.cc:344] Connected to a master server at 127.13.88.190:41231
I20260128 15:02:01.459928 13914 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:01.460402 13914 heartbeater.cc:507] Master 127.13.88.190:41231 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.130@KRBTEST.COM" created.
I20260128 15:02:01.461858 13721 ts_manager.cc:194] Registered new tserver with Master: 14a05d66fcc248a5b0d4d4c953778345 (127.13.88.129:36311)
I20260128 15:02:01.463056 13721 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.129@KRBTEST.COM'} at 127.13.88.129:54011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:01.490908 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:41231
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:01.592743 13921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:01.592968 13921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:01.593046 13921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:01.596217 13921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:01.596276 13921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:01.596376 13921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:02:01.600363 13921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:41231
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:01.601325 13921 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:01.602093 13921 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:01.607947 13929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:01.607935 13926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:01.607926 13927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:01.608258 13921 server_base.cc:1060] running on GCE node
I20260128 15:02:01.608595 13921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:01.609061 13921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:01.610196 13921 hybrid_clock.cc:648] HybridClock initialized: now 1769612521610169 us; error 42 us; skew 500 ppm
Jan 28 15:02:01 dist-test-slave-m28t krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612521, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:01.612648 13921 init.cc:377] Logged in from keytab as kudu/127.13.88.130@KRBTEST.COM (short username kudu)
I20260128 15:02:01.613592 13921 webserver.cc:492] Webserver started at http://127.13.88.130:35843/ using document root <none> and password file <none>
I20260128 15:02:01.614080 13921 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:01.614128 13921 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:01.614290 13921 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:01.615923 13921 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "31294a792e844f23a427bad011b9fb8b"
format_stamp: "Formatted at 2026-01-28 15:02:01 on dist-test-slave-m28t"
server_key: "c82391690de8097af39a8dae9f7f3b44"
server_key_iv: "0cf30b01baba9d52cdfe2528fdc60c20"
server_key_version: "encryptionkey@0"
I20260128 15:02:01.616371 13921 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "31294a792e844f23a427bad011b9fb8b"
format_stamp: "Formatted at 2026-01-28 15:02:01 on dist-test-slave-m28t"
server_key: "c82391690de8097af39a8dae9f7f3b44"
server_key_iv: "0cf30b01baba9d52cdfe2528fdc60c20"
server_key_version: "encryptionkey@0"
I20260128 15:02:01.619474 13921 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260128 15:02:01.621438 13936 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:01.622311 13921 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:02:01.622416 13921 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "31294a792e844f23a427bad011b9fb8b"
format_stamp: "Formatted at 2026-01-28 15:02:01 on dist-test-slave-m28t"
server_key: "c82391690de8097af39a8dae9f7f3b44"
server_key_iv: "0cf30b01baba9d52cdfe2528fdc60c20"
server_key_version: "encryptionkey@0"
I20260128 15:02:01.622493 13921 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:01.632534 13921 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:01.634711 13921 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:01.634857 13921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:01.635324 13921 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:01.636091 13921 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:01.636137 13921 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:01.636183 13921 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:01.636210 13921 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:01.644205 13921 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:43707
I20260128 15:02:01.644229 14049 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:43707 every 8 connection(s)
I20260128 15:02:01.645085 13921 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
I20260128 15:02:01.645368 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 13921
I20260128 15:02:01.645465 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:02:01.645684 13666 external_mini_cluster.cc:1468] Setting key e209bb4327c22350d9b0a784b555116e
Jan 28 15:02:01 dist-test-slave-m28t krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612521, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.131@KRBTEST.COM" created.
I20260128 15:02:01.657406 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:01.646726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:44777 (local address 127.13.88.190:41231)
0128 15:02:01.646833 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:01.646836 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:01.647673 (+ 837us) server_negotiation.cc:408] Connection header received
0128 15:02:01.648649 (+ 976us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:01.648653 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:01.648708 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:01.648796 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:01.650577 (+ 1781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.651080 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:01.651937 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.652063 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:01.654445 (+ 2382us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:01.654462 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:01.654464 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:01.654489 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:01.655668 (+ 1179us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:01.656114 (+ 446us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:01.656116 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:01.656117 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:01.656154 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:01.656369 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:01.656372 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:01.656373 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:01.656544 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:01.656619 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:01.657117 (+ 498us) server_negotiation.cc:300] Negotiation successful
0128 15:02:01.657237 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260128 15:02:01.657945 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:01.647076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41231 (local address 127.13.88.130:44777)
0128 15:02:01.647526 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:01.647571 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:01.648457 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:01.648953 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:01.648960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:01.649299 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:01.650418 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:01.650435 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.651200 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:01.651204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:01.651829 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:01.651839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.652052 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:01.652624 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:01.652659 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:01.654304 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:01.655771 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:01.655776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:01.655784 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:01.656029 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:01.656213 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:01.656215 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:01.656216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:01.656292 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:01.656623 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:01.656629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:01.656864 (+ 235us) client_negotiation.cc:769] Sending connection context
0128 15:02:01.657079 (+ 215us) client_negotiation.cc:240] Negotiation successful
0128 15:02:01.657285 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":117,"threads_started":1}
I20260128 15:02:01.658921 14050 heartbeater.cc:344] Connected to a master server at 127.13.88.190:41231
I20260128 15:02:01.659139 14050 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:01.659636 14050 heartbeater.cc:507] Master 127.13.88.190:41231 requested a full tablet report, sending...
I20260128 15:02:01.660632 13721 ts_manager.cc:194] Registered new tserver with Master: 31294a792e844f23a427bad011b9fb8b (127.13.88.130:43707)
I20260128 15:02:01.661154 13721 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.130@KRBTEST.COM'} at 127.13.88.130:44777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:01.686893 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:41231
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:01.788082 14057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:01.788282 14057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:01.788337 14057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:01.791514 14057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:01.791571 14057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:01.791654 14057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:02:01.795637 14057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:41231
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:01.796588 14057 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:01.797353 14057 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:01.803499 14062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:01.803536 14065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:01.803632 14063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:01.803795 14057 server_base.cc:1060] running on GCE node
I20260128 15:02:01.804117 14057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:01.804654 14057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:01.805787 14057 hybrid_clock.cc:648] HybridClock initialized: now 1769612521805776 us; error 27 us; skew 500 ppm
Jan 28 15:02:01 dist-test-slave-m28t krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612521, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:01.808133 14057 init.cc:377] Logged in from keytab as kudu/127.13.88.131@KRBTEST.COM (short username kudu)
I20260128 15:02:01.809092 14057 webserver.cc:492] Webserver started at http://127.13.88.131:40139/ using document root <none> and password file <none>
I20260128 15:02:01.809578 14057 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:01.809621 14057 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:01.809772 14057 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:01.811388 14057 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "90fddaa61ee542769b11f68895eace8e"
format_stamp: "Formatted at 2026-01-28 15:02:01 on dist-test-slave-m28t"
server_key: "f5c8944e6477ccbe1521fb67804891ad"
server_key_iv: "f0acddb0799d4c9f4038efd327b72d2d"
server_key_version: "encryptionkey@0"
I20260128 15:02:01.811798 14057 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "90fddaa61ee542769b11f68895eace8e"
format_stamp: "Formatted at 2026-01-28 15:02:01 on dist-test-slave-m28t"
server_key: "f5c8944e6477ccbe1521fb67804891ad"
server_key_iv: "f0acddb0799d4c9f4038efd327b72d2d"
server_key_version: "encryptionkey@0"
I20260128 15:02:01.814936 14057 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260128 15:02:01.816942 14072 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:01.817867 14057 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:01.817956 14057 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "90fddaa61ee542769b11f68895eace8e"
format_stamp: "Formatted at 2026-01-28 15:02:01 on dist-test-slave-m28t"
server_key: "f5c8944e6477ccbe1521fb67804891ad"
server_key_iv: "f0acddb0799d4c9f4038efd327b72d2d"
server_key_version: "encryptionkey@0"
I20260128 15:02:01.818042 14057 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:01.845062 14057 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:01.851339 14057 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:01.851527 14057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:01.852068 14057 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:01.853014 14057 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:01.853081 14057 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:01.853135 14057 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:01.853158 14057 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:01.861805 14057 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:34711
I20260128 15:02:01.861831 14185 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:34711 every 8 connection(s)
I20260128 15:02:01.862694 14057 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
Jan 28 15:02:01 dist-test-slave-m28t krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612521, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:01.871842 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 14057
I20260128 15:02:01.871922 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:02:01.872107 13666 external_mini_cluster.cc:1468] Setting key dfe2be644e5de6943f0bd14daa62bb87
I20260128 15:02:01.874081 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:01.864247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:35447 (local address 127.13.88.190:41231)
0128 15:02:01.864384 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:01.864388 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:01.865066 (+ 678us) server_negotiation.cc:408] Connection header received
0128 15:02:01.865854 (+ 788us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:01.865857 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:01.865909 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:01.865979 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:01.867588 (+ 1609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.868140 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:01.868809 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.868946 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:01.870918 (+ 1972us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:01.870938 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:01.870940 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:01.870967 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:01.872126 (+ 1159us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:01.872639 (+ 513us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:01.872642 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:01.872643 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:01.872694 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:01.873013 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:01.873017 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:01.873019 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:01.873188 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:01.873287 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:01.873814 (+ 527us) server_negotiation.cc:300] Negotiation successful
0128 15:02:01.873934 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260128 15:02:01.874641 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:01.864495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41231 (local address 127.13.88.131:35447)
0128 15:02:01.864920 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:01.864957 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:01.865671 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:01.866117 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:01.866125 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:01.866503 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:01.867445 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:01.867460 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.868233 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:01.868235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:01.868718 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:01.868727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.868907 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:01.869397 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:01.869421 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:01.870773 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:01.872229 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:01.872235 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:01.872247 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:01.872532 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:01.872781 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:01.872785 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:01.872786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:01.872905 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:01.873283 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:01.873288 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:01.873538 (+ 250us) client_negotiation.cc:769] Sending connection context
0128 15:02:01.873739 (+ 201us) client_negotiation.cc:240] Negotiation successful
0128 15:02:01.873954 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
I20260128 15:02:01.875577 14186 heartbeater.cc:344] Connected to a master server at 127.13.88.190:41231
I20260128 15:02:01.875793 14186 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:01.876257 14186 heartbeater.cc:507] Master 127.13.88.190:41231 requested a full tablet report, sending...
I20260128 15:02:01.877137 13721 ts_manager.cc:194] Registered new tserver with Master: 90fddaa61ee542769b11f68895eace8e (127.13.88.131:34711)
I20260128 15:02:01.877647 13721 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.131@KRBTEST.COM'} at 127.13.88.131:35447
I20260128 15:02:01.885919 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 28 15:02:01 dist-test-slave-m28t krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612521, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 28 15:02:01 dist-test-slave-m28t krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612521, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.88.129@KRBTEST.COM
I20260128 15:02:01.907486 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:01.900160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.13.88.129:36311)
0128 15:02:01.900490 (+ 330us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:01.900496 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:01.900521 (+ 25us) server_negotiation.cc:408] Connection header received
0128 15:02:01.900570 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:01.900575 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:01.900751 (+ 176us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:01.900867 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:01.901568 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.902258 (+ 690us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:01.902840 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.903013 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:01.904863 (+ 1850us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:01.904884 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:01.904896 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:01.904919 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:01.906245 (+ 1326us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:01.906575 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:01.906580 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:01.906581 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:01.906633 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:01.906782 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:01.906784 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:01.906785 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:01.906945 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:01.907096 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:01.907192 (+ 96us) server_negotiation.cc:300] Negotiation successful
0128 15:02:01.907307 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":114,"threads_started":1}
W20260128 15:02:01.908329 13828 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35230
Jan 28 15:02:01 dist-test-slave-m28t krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 28 15:02:01 dist-test-slave-m28t krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.129@KRBTEST.COM
I20260128 15:02:01.929742 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:01.923014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.13.88.129:36311)
0128 15:02:01.923175 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:01.923178 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:01.923259 (+ 81us) server_negotiation.cc:408] Connection header received
0128 15:02:01.923350 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:01.923353 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:01.923399 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:01.923468 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:01.924138 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.924682 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:01.925255 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:01.925410 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:01.927298 (+ 1888us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:01.927314 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:01.927316 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:01.927339 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:01.928453 (+ 1114us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:01.928796 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:01.928800 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:01.928802 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:01.928845 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:01.929034 (+ 189us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:01.929038 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:01.929040 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:01.929168 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:01.929280 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:01.929479 (+ 199us) server_negotiation.cc:300] Negotiation successful
0128 15:02:01.929588 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260128 15:02:01.930619 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 13785
I20260128 15:02:01.936215 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 13921
I20260128 15:02:01.942793 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 14057
I20260128 15:02:01.947366 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 13689
2026-01-28T15:02:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2907 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:02:01 dist-test-slave-m28t krb5kdc[14200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:02:01 dist-test-slave-m28t krb5kdc[14200](info): set up 2 sockets
Jan 28 15:02:01 dist-test-slave-m28t krb5kdc[14200](info): commencing operation
krb5kdc: starting...
W20260128 15:02:03.779999 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.812s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Jan 28 15:02:03 dist-test-slave-m28t krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:02:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:03.900375 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:38449
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:38449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:04.003105 14216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:04.003315 14216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:04.003381 14216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:04.006604 14216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:04.006662 14216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:04.006686 14216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:04.006718 14216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:04.006740 14216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:02:04.010828 14216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:38449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:38449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:04.011756 14216 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:04.012560 14216 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:04.017704 14222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:04.017715 14224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:04.017715 14221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:04.017904 14216 server_base.cc:1060] running on GCE node
I20260128 15:02:04.018481 14216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:04.019292 14216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:04.020450 14216 hybrid_clock.cc:648] HybridClock initialized: now 1769612524020436 us; error 26 us; skew 500 ppm
Jan 28 15:02:04 dist-test-slave-m28t krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612524, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:04.022735 14216 init.cc:377] Logged in from keytab as kudu/127.13.88.190@KRBTEST.COM (short username kudu)
I20260128 15:02:04.023690 14216 webserver.cc:492] Webserver started at http://127.13.88.190:35401/ using document root <none> and password file <none>
I20260128 15:02:04.024170 14216 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:04.024217 14216 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:04.024399 14216 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:04.026008 14216 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "e5ad8e78e7c843a1ae67de8e0d43b31b"
format_stamp: "Formatted at 2026-01-28 15:02:04 on dist-test-slave-m28t"
server_key: "d51f050898eb6318b3a618ac8a95a9f2"
server_key_iv: "22802f174d194dfd262da57e82c2ca00"
server_key_version: "encryptionkey@0"
I20260128 15:02:04.026448 14216 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "e5ad8e78e7c843a1ae67de8e0d43b31b"
format_stamp: "Formatted at 2026-01-28 15:02:04 on dist-test-slave-m28t"
server_key: "d51f050898eb6318b3a618ac8a95a9f2"
server_key_iv: "22802f174d194dfd262da57e82c2ca00"
server_key_version: "encryptionkey@0"
I20260128 15:02:04.029489 14216 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260128 15:02:04.031374 14231 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:04.032262 14216 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:04.032359 14216 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "e5ad8e78e7c843a1ae67de8e0d43b31b"
format_stamp: "Formatted at 2026-01-28 15:02:04 on dist-test-slave-m28t"
server_key: "d51f050898eb6318b3a618ac8a95a9f2"
server_key_iv: "22802f174d194dfd262da57e82c2ca00"
server_key_version: "encryptionkey@0"
I20260128 15:02:04.032474 14216 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:04.052059 14216 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:04.054473 14216 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:04.054646 14216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:04.061254 14283 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:38449 every 8 connection(s)
I20260128 15:02:04.061259 14216 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:38449
I20260128 15:02:04.062232 14216 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:04.064985 14284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:04.064986 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 14216
I20260128 15:02:04.065095 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:04.065335 13666 external_mini_cluster.cc:1468] Setting key ff352f22b2c14932998c3286a0bf83d8
I20260128 15:02:04.070804 14284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b: Bootstrap starting.
Jan 28 15:02:04 dist-test-slave-m28t krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:04.073192 14284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:04.073836 14284 log.cc:826] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:04.075439 14284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b: No bootstrap required, opened a new log
I20260128 15:02:04.077119 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.066478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.13.88.190:38449)
0128 15:02:04.066828 (+ 350us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:04.066838 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:04.066870 (+ 32us) server_negotiation.cc:408] Connection header received
0128 15:02:04.067783 (+ 913us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:04.067802 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:04.068072 (+ 270us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:04.068366 (+ 294us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:04.069310 (+ 944us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.070048 (+ 738us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.070859 (+ 811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.071095 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.073290 (+ 2195us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:04.073308 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:04.073322 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:04.073355 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:04.074835 (+ 1480us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:04.075197 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:04.075202 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:04.075206 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:04.075277 (+ 71us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:04.075497 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:04.075500 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:04.075502 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:04.075872 (+ 370us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:04.076037 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:04.076308 (+ 271us) server_negotiation.cc:300] Negotiation successful
0128 15:02:04.076560 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":101,"threads_started":1}
I20260128 15:02:04.077971 14284 raft_consensus.cc:359] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ad8e78e7c843a1ae67de8e0d43b31b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 38449 } }
I20260128 15:02:04.078146 14284 raft_consensus.cc:385] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:04.078212 14284 raft_consensus.cc:740] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ad8e78e7c843a1ae67de8e0d43b31b, State: Initialized, Role: FOLLOWER
I20260128 15:02:04.078596 14284 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ad8e78e7c843a1ae67de8e0d43b31b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 38449 } }
I20260128 15:02:04.078718 14284 raft_consensus.cc:399] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:04.078776 14284 raft_consensus.cc:493] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:04.078860 14284 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:04.079648 14284 raft_consensus.cc:515] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ad8e78e7c843a1ae67de8e0d43b31b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 38449 } }
I20260128 15:02:04.079964 14284 leader_election.cc:304] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5ad8e78e7c843a1ae67de8e0d43b31b; no voters:
I20260128 15:02:04.080205 14284 leader_election.cc:290] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:04.080284 14289 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:04.080504 14289 raft_consensus.cc:697] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [term 1 LEADER]: Becoming Leader. State: Replica: e5ad8e78e7c843a1ae67de8e0d43b31b, State: Running, Role: LEADER
I20260128 15:02:04.080802 14289 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ad8e78e7c843a1ae67de8e0d43b31b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 38449 } }
I20260128 15:02:04.081236 14284 sys_catalog.cc:565] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:04.082291 14290 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5ad8e78e7c843a1ae67de8e0d43b31b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ad8e78e7c843a1ae67de8e0d43b31b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 38449 } } }
I20260128 15:02:04.082407 14290 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:04.082811 14291 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5ad8e78e7c843a1ae67de8e0d43b31b. Latest consensus state: current_term: 1 leader_uuid: "e5ad8e78e7c843a1ae67de8e0d43b31b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ad8e78e7c843a1ae67de8e0d43b31b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 38449 } } }
I20260128 15:02:04.082880 14298 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:04.082967 14291 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:04.085830 14298 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:04.090787 14298 catalog_manager.cc:1357] Generated new cluster ID: 87b0e3754850449b82698c8d3afaebc9
I20260128 15:02:04.090845 14298 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:04.098176 14298 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:04.099282 14298 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:04.106684 14298 catalog_manager.cc:6027] T 00000000000000000000000000000000 P e5ad8e78e7c843a1ae67de8e0d43b31b: Generated new TSK 0
I20260128 15:02:04.107419 14298 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:04.158545 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:38449
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:04.260782 14312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:04.260995 14312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:04.261075 14312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:04.264220 14312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:04.264277 14312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:04.264385 14312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:02:04.268406 14312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:38449
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:04.269418 14312 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:04.270217 14312 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:04.276203 14320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:04.276269 14318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:04.276235 14317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:04.276537 14312 server_base.cc:1060] running on GCE node
I20260128 15:02:04.276839 14312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:04.277354 14312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:04.278514 14312 hybrid_clock.cc:648] HybridClock initialized: now 1769612524278489 us; error 37 us; skew 500 ppm
Jan 28 15:02:04 dist-test-slave-m28t krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612524, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:04.280906 14312 init.cc:377] Logged in from keytab as kudu/127.13.88.129@KRBTEST.COM (short username kudu)
I20260128 15:02:04.281898 14312 webserver.cc:492] Webserver started at http://127.13.88.129:42001/ using document root <none> and password file <none>
I20260128 15:02:04.282473 14312 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:04.282527 14312 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:04.282706 14312 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:04.284543 14312 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "d5d84f5fc4c34fe0b8feb595f74267cc"
format_stamp: "Formatted at 2026-01-28 15:02:04 on dist-test-slave-m28t"
server_key: "cbf11008d645de28b47ccb14f0540eaa"
server_key_iv: "430658cf98ef49533562ea5f9cdfbb3c"
server_key_version: "encryptionkey@0"
I20260128 15:02:04.285027 14312 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "d5d84f5fc4c34fe0b8feb595f74267cc"
format_stamp: "Formatted at 2026-01-28 15:02:04 on dist-test-slave-m28t"
server_key: "cbf11008d645de28b47ccb14f0540eaa"
server_key_iv: "430658cf98ef49533562ea5f9cdfbb3c"
server_key_version: "encryptionkey@0"
I20260128 15:02:04.288208 14312 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260128 15:02:04.290292 14327 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:04.291221 14312 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:04.291312 14312 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "d5d84f5fc4c34fe0b8feb595f74267cc"
format_stamp: "Formatted at 2026-01-28 15:02:04 on dist-test-slave-m28t"
server_key: "cbf11008d645de28b47ccb14f0540eaa"
server_key_iv: "430658cf98ef49533562ea5f9cdfbb3c"
server_key_version: "encryptionkey@0"
I20260128 15:02:04.291404 14312 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:04.313917 14312 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:04.316567 14312 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:04.316725 14312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:04.317214 14312 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:04.318024 14312 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:04.318068 14312 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:04.318109 14312 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:04.318135 14312 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:04.326275 14312 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:35585
I20260128 15:02:04.326305 14440 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:35585 every 8 connection(s)
I20260128 15:02:04.327129 14312 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:02:04.333762 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 14312
I20260128 15:02:04.333844 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:02:04.334057 13666 external_mini_cluster.cc:1468] Setting key e1db3a22fc6ff4029e56e13eda7e2480
Jan 28 15:02:04 dist-test-slave-m28t krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612524, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:04.338299 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.328691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:57827 (local address 127.13.88.190:38449)
0128 15:02:04.328808 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:04.328811 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:04.329442 (+ 631us) server_negotiation.cc:408] Connection header received
0128 15:02:04.330170 (+ 728us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:04.330173 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:04.330223 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:04.330294 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:04.331771 (+ 1477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.332270 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.332914 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.333043 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.335098 (+ 2055us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:04.335116 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:04.335119 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:04.335143 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:04.336405 (+ 1262us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:04.336962 (+ 557us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:04.336965 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:04.336966 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:04.337013 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:04.337286 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:04.337288 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:04.337290 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:04.337432 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:04.337499 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:04.338070 (+ 571us) server_negotiation.cc:300] Negotiation successful
0128 15:02:04.338179 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260128 15:02:04.339118 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.328905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:38449 (local address 127.13.88.129:57827)
0128 15:02:04.329282 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:04.329322 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:04.330008 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:04.330426 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:04.330433 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:04.330773 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:04.331654 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:04.331667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.332384 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.332387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:04.332823 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:04.332830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.332989 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.333472 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:04.333496 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:04.334952 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:04.336534 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:04.336540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:04.336552 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:04.336869 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:04.337097 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:04.337100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:04.337103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:04.337212 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:04.337517 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:04.337524 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:04.337819 (+ 295us) client_negotiation.cc:769] Sending connection context
0128 15:02:04.338026 (+ 207us) client_negotiation.cc:240] Negotiation successful
0128 15:02:04.338290 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":96,"threads_started":1}
I20260128 15:02:04.340251 14441 heartbeater.cc:344] Connected to a master server at 127.13.88.190:38449
I20260128 15:02:04.340512 14441 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:04.341007 14441 heartbeater.cc:507] Master 127.13.88.190:38449 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.88.130@KRBTEST.COM; defaulting to no policy
I20260128 15:02:04.342394 14248 ts_manager.cc:194] Registered new tserver with Master: d5d84f5fc4c34fe0b8feb595f74267cc (127.13.88.129:35585)
I20260128 15:02:04.343667 14248 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.129@KRBTEST.COM'} at 127.13.88.129:57827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:04.376477 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:38449
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:04.476351 14448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:04.476581 14448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:04.476660 14448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:04.479794 14448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:04.479851 14448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:04.479943 14448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:02:04.483994 14448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:38449
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:04.484959 14448 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:04.485738 14448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:04.491744 14454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:04.491766 14453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:04.491910 14448 server_base.cc:1060] running on GCE node
W20260128 15:02:04.491771 14456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:04.492357 14448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:04.492890 14448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:04.494027 14448 hybrid_clock.cc:648] HybridClock initialized: now 1769612524494009 us; error 34 us; skew 500 ppm
Jan 28 15:02:04 dist-test-slave-m28t krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612524, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:04.496402 14448 init.cc:377] Logged in from keytab as kudu/127.13.88.130@KRBTEST.COM (short username kudu)
I20260128 15:02:04.497373 14448 webserver.cc:492] Webserver started at http://127.13.88.130:40229/ using document root <none> and password file <none>
I20260128 15:02:04.497861 14448 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:04.497905 14448 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:04.498047 14448 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:04.499586 14448 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "79cc4c0a72de4285aabf74e324639cd0"
format_stamp: "Formatted at 2026-01-28 15:02:04 on dist-test-slave-m28t"
server_key: "ef7cc2382a5b501a6280769d670d9c3f"
server_key_iv: "a05727dc3d917d58c8ce4aff2e50d4c4"
server_key_version: "encryptionkey@0"
I20260128 15:02:04.499989 14448 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "79cc4c0a72de4285aabf74e324639cd0"
format_stamp: "Formatted at 2026-01-28 15:02:04 on dist-test-slave-m28t"
server_key: "ef7cc2382a5b501a6280769d670d9c3f"
server_key_iv: "a05727dc3d917d58c8ce4aff2e50d4c4"
server_key_version: "encryptionkey@0"
I20260128 15:02:04.503057 14448 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260128 15:02:04.505007 14463 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:04.505915 14448 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:04.505999 14448 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "79cc4c0a72de4285aabf74e324639cd0"
format_stamp: "Formatted at 2026-01-28 15:02:04 on dist-test-slave-m28t"
server_key: "ef7cc2382a5b501a6280769d670d9c3f"
server_key_iv: "a05727dc3d917d58c8ce4aff2e50d4c4"
server_key_version: "encryptionkey@0"
I20260128 15:02:04.506069 14448 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:04.524370 14448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:04.527297 14448 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:04.527444 14448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:04.527897 14448 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:04.528739 14448 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:04.528787 14448 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:04.528824 14448 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:04.528844 14448 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:04.536769 14448 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:44165
I20260128 15:02:04.536834 14576 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:44165 every 8 connection(s)
I20260128 15:02:04.537639 14448 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
I20260128 15:02:04.540902 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 14448
I20260128 15:02:04.541009 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:02:04.541245 13666 external_mini_cluster.cc:1468] Setting key c556e81200717a3048aa5cb74d27b615
Jan 28 15:02:04 dist-test-slave-m28t krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612524, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:04.549188 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.539138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:54301 (local address 127.13.88.190:38449)
0128 15:02:04.539252 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:04.539255 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:04.539888 (+ 633us) server_negotiation.cc:408] Connection header received
0128 15:02:04.540679 (+ 791us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:04.540683 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:04.540734 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:04.540804 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:04.542646 (+ 1842us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.543126 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.543903 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.544031 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.546212 (+ 2181us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:04.546227 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:04.546229 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:04.546255 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:04.547489 (+ 1234us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:04.547928 (+ 439us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:04.547932 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:04.547934 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:04.547978 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:04.548217 (+ 239us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:04.548220 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:04.548222 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:04.548362 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:04.548479 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:04.548915 (+ 436us) server_negotiation.cc:300] Negotiation successful
0128 15:02:04.549034 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
WARNING: no policy specified for kudu/127.13.88.131@KRBTEST.COM; defaulting to no policy
I20260128 15:02:04.549753 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.539368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:38449 (local address 127.13.88.130:54301)
0128 15:02:04.539756 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:04.539788 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:04.540511 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:04.540981 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:04.540993 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:04.541476 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:04.542507 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:04.542525 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.543240 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.543243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:04.543801 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:04.543810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.544018 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.544589 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:04.544620 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:04.546076 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:04.547597 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:04.547602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:04.547612 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:04.547838 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:04.548053 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:04.548056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:04.548058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:04.548137 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:04.548473 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:04.548479 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:04.548695 (+ 216us) client_negotiation.cc:769] Sending connection context
0128 15:02:04.548865 (+ 170us) client_negotiation.cc:240] Negotiation successful
0128 15:02:04.549107 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":90,"threads_started":1}
I20260128 15:02:04.550742 14577 heartbeater.cc:344] Connected to a master server at 127.13.88.190:38449
I20260128 15:02:04.550971 14577 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.131@KRBTEST.COM" created.
I20260128 15:02:04.551514 14577 heartbeater.cc:507] Master 127.13.88.190:38449 requested a full tablet report, sending...
I20260128 15:02:04.552515 14248 ts_manager.cc:194] Registered new tserver with Master: 79cc4c0a72de4285aabf74e324639cd0 (127.13.88.130:44165)
I20260128 15:02:04.553054 14248 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.130@KRBTEST.COM'} at 127.13.88.130:54301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:04.580068 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:38449
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:39893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:04.680352 14584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:04.680578 14584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:04.680644 14584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:04.683805 14584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:04.683854 14584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:04.683928 14584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:02:04.687803 14584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:39893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:38449
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:04.688726 14584 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:04.689451 14584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:04.695312 14592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:04.695284 14590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:04.695266 14589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:04.695394 14584 server_base.cc:1060] running on GCE node
I20260128 15:02:04.695919 14584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:04.696363 14584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:04.697513 14584 hybrid_clock.cc:648] HybridClock initialized: now 1769612524697499 us; error 28 us; skew 500 ppm
Jan 28 15:02:04 dist-test-slave-m28t krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612524, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:04.699815 14584 init.cc:377] Logged in from keytab as kudu/127.13.88.131@KRBTEST.COM (short username kudu)
I20260128 15:02:04.700788 14584 webserver.cc:492] Webserver started at http://127.13.88.131:37851/ using document root <none> and password file <none>
I20260128 15:02:04.701268 14584 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:04.701314 14584 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:04.701474 14584 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:04.703013 14584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "32bba4052eeb44f4ac324e014cd9d426"
format_stamp: "Formatted at 2026-01-28 15:02:04 on dist-test-slave-m28t"
server_key: "1c16c7b98ee558eba1bf275dcf80a94a"
server_key_iv: "c01f101bef4344a6d8517eb2c42e7505"
server_key_version: "encryptionkey@0"
I20260128 15:02:04.703431 14584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "32bba4052eeb44f4ac324e014cd9d426"
format_stamp: "Formatted at 2026-01-28 15:02:04 on dist-test-slave-m28t"
server_key: "1c16c7b98ee558eba1bf275dcf80a94a"
server_key_iv: "c01f101bef4344a6d8517eb2c42e7505"
server_key_version: "encryptionkey@0"
I20260128 15:02:04.706458 14584 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260128 15:02:04.708308 14599 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:04.709237 14584 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:04.709332 14584 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "32bba4052eeb44f4ac324e014cd9d426"
format_stamp: "Formatted at 2026-01-28 15:02:04 on dist-test-slave-m28t"
server_key: "1c16c7b98ee558eba1bf275dcf80a94a"
server_key_iv: "c01f101bef4344a6d8517eb2c42e7505"
server_key_version: "encryptionkey@0"
I20260128 15:02:04.709406 14584 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:04.723489 14584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:04.726075 14584 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:04.726217 14584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:04.726699 14584 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:04.727489 14584 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:04.727535 14584 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:04.727583 14584 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:04.727608 14584 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:04.735587 14584 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:40605
I20260128 15:02:04.735620 14712 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:40605 every 8 connection(s)
I20260128 15:02:04.736452 14584 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
I20260128 15:02:04.744329 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 14584
I20260128 15:02:04.744458 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:02:04.744669 13666 external_mini_cluster.cc:1468] Setting key 363ced93a4cf72c18b950d77e5aa8360
Jan 28 15:02:04 dist-test-slave-m28t krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612524, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:04.750511 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.737982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:55263 (local address 127.13.88.190:38449)
0128 15:02:04.738113 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:04.738117 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:04.738715 (+ 598us) server_negotiation.cc:408] Connection header received
0128 15:02:04.739504 (+ 789us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:04.739508 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:04.739565 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:04.739636 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:04.741196 (+ 1560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.741748 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.742502 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.742689 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.747133 (+ 4444us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:04.747152 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:04.747155 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:04.747186 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:04.748753 (+ 1567us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:04.749221 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:04.749226 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:04.749228 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:04.749273 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:04.749586 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:04.749590 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:04.749592 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:04.749778 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:04.749877 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:04.750290 (+ 413us) server_negotiation.cc:300] Negotiation successful
0128 15:02:04.750399 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260128 15:02:04.751109 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.738204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:38449 (local address 127.13.88.131:55263)
0128 15:02:04.738578 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:04.738610 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:04.739310 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:04.739798 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:04.739806 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:04.740155 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:04.741048 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:04.741062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.741864 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.741866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:04.742397 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:04.742408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.742610 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.745162 (+ 2552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:04.745196 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:04.746984 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:04.748862 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:04.748868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:04.748881 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:04.749132 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:04.749370 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:04.749374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:04.749376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:04.749503 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:04.749872 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:04.749876 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:04.750087 (+ 211us) client_negotiation.cc:769] Sending connection context
0128 15:02:04.750247 (+ 160us) client_negotiation.cc:240] Negotiation successful
0128 15:02:04.750457 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":95,"threads_started":1}
I20260128 15:02:04.752025 14713 heartbeater.cc:344] Connected to a master server at 127.13.88.190:38449
I20260128 15:02:04.752231 14713 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:04.752782 14713 heartbeater.cc:507] Master 127.13.88.190:38449 requested a full tablet report, sending...
I20260128 15:02:04.753644 14248 ts_manager.cc:194] Registered new tserver with Master: 32bba4052eeb44f4ac324e014cd9d426 (127.13.88.131:40605)
I20260128 15:02:04.754165 14248 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.131@KRBTEST.COM'} at 127.13.88.131:55263
I20260128 15:02:04.758353 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260128 15:02:04.766376 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.760902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.13.88.190:38449)
0128 15:02:04.761021 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:04.761025 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:04.761161 (+ 136us) server_negotiation.cc:408] Connection header received
0128 15:02:04.761248 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:04.761251 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:04.761287 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:04.761347 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:04.762061 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.762519 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.763060 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.763176 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.764053 (+ 877us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:04.764065 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:04.764067 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:04.764087 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:04.765352 (+ 1265us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:04.765651 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:04.765653 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:04.765655 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:04.765689 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:04.765820 (+ 131us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:04.765822 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:04.765823 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:04.765969 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:04.766054 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:04.766181 (+ 127us) server_negotiation.cc:300] Negotiation successful
0128 15:02:04.766262 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260128 15:02:04.769788 14248 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 15:02:04.772006 14248 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:02:04.783885 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.779827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:35585 (local address 127.0.0.1:35408)
0128 15:02:04.780159 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:04.780195 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:04.780307 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:04.781192 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:04.781195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:04.781209 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:04.781445 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:04.781453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.782471 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.782475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:04.783353 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:04.783363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.783469 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.783523 (+ 54us) client_negotiation.cc:769] Sending connection context
0128 15:02:04.783603 (+ 80us) client_negotiation.cc:240] Negotiation successful
0128 15:02:04.783692 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":95,"threads_started":1}
I20260128 15:02:04.784727 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.780098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50664 (local address 127.13.88.131:40605)
0128 15:02:04.780507 (+ 409us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:04.780513 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:04.780535 (+ 22us) server_negotiation.cc:408] Connection header received
0128 15:02:04.780654 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:04.780660 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:04.780836 (+ 176us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:04.781415 (+ 579us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:04.781544 (+ 129us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.782551 (+ 1007us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.783813 (+ 1262us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.784291 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.784408 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:04.784500 (+ 92us) server_negotiation.cc:300] Negotiation successful
0128 15:02:04.784586 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":152,"threads_started":1}
I20260128 15:02:04.784724 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.780286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35408 (local address 127.13.88.129:35585)
0128 15:02:04.780780 (+ 494us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:04.780788 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:04.780810 (+ 22us) server_negotiation.cc:408] Connection header received
0128 15:02:04.780893 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:04.780899 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:04.781066 (+ 167us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:04.781201 (+ 135us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:04.781560 (+ 359us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.782365 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.783778 (+ 1413us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.784363 (+ 585us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.784486 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:04.784536 (+ 50us) server_negotiation.cc:300] Negotiation successful
0128 15:02:04.784605 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":97,"threads_started":1}
I20260128 15:02:04.784914 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.779506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.13.88.130:44165)
0128 15:02:04.779854 (+ 348us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:04.779860 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:04.780841 (+ 981us) server_negotiation.cc:408] Connection header received
0128 15:02:04.781039 (+ 198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:04.781045 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:04.781184 (+ 139us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:04.781277 (+ 93us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:04.781774 (+ 497us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.782849 (+ 1075us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.784179 (+ 1330us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.784582 (+ 403us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.784666 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:04.784716 (+ 50us) server_negotiation.cc:300] Negotiation successful
0128 15:02:04.784777 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":94,"threads_started":1}
I20260128 15:02:04.784950 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.779543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:40605 (local address 127.0.0.1:50664)
0128 15:02:04.780410 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:04.780446 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:04.780530 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:04.781004 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:04.781010 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:04.781033 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:04.781281 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:04.781290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.782687 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.782691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:04.783501 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:04.783509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.783594 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.783609 (+ 15us) client_negotiation.cc:769] Sending connection context
0128 15:02:04.783660 (+ 51us) client_negotiation.cc:240] Negotiation successful
0128 15:02:04.783701 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":107,"threads_started":1}
I20260128 15:02:04.785243 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.779377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:44165 (local address 127.0.0.1:38294)
0128 15:02:04.780759 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:04.780772 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:04.780906 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:04.781428 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:04.781431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:04.781446 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:04.781658 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:04.781666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.782946 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.782950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:04.783749 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:04.783758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.783851 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.783866 (+ 15us) client_negotiation.cc:769] Sending connection context
0128 15:02:04.783909 (+ 43us) client_negotiation.cc:240] Negotiation successful
0128 15:02:04.783963 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":78,"threads_started":1}
I20260128 15:02:04.787254 14647 tablet_service.cc:1505] Processing CreateTablet for tablet e2257fcb8e1f4a829f1aa7815c588e5a (DEFAULT_TABLE table=test-table [id=3c4f3149aab44c79b9ac7fa8e150c102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:04.787371 14375 tablet_service.cc:1505] Processing CreateTablet for tablet e2257fcb8e1f4a829f1aa7815c588e5a (DEFAULT_TABLE table=test-table [id=3c4f3149aab44c79b9ac7fa8e150c102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:04.787372 14511 tablet_service.cc:1505] Processing CreateTablet for tablet e2257fcb8e1f4a829f1aa7815c588e5a (DEFAULT_TABLE table=test-table [id=3c4f3149aab44c79b9ac7fa8e150c102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:04.788069 14647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2257fcb8e1f4a829f1aa7815c588e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:04.788221 14375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2257fcb8e1f4a829f1aa7815c588e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:04.788221 14511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2257fcb8e1f4a829f1aa7815c588e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:04.793341 14730 tablet_bootstrap.cc:492] T e2257fcb8e1f4a829f1aa7815c588e5a P 32bba4052eeb44f4ac324e014cd9d426: Bootstrap starting.
I20260128 15:02:04.794430 14731 tablet_bootstrap.cc:492] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc: Bootstrap starting.
I20260128 15:02:04.795111 14732 tablet_bootstrap.cc:492] T e2257fcb8e1f4a829f1aa7815c588e5a P 79cc4c0a72de4285aabf74e324639cd0: Bootstrap starting.
I20260128 15:02:04.795567 14730 tablet_bootstrap.cc:654] T e2257fcb8e1f4a829f1aa7815c588e5a P 32bba4052eeb44f4ac324e014cd9d426: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:04.796458 14730 log.cc:826] T e2257fcb8e1f4a829f1aa7815c588e5a P 32bba4052eeb44f4ac324e014cd9d426: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:04.796555 14731 tablet_bootstrap.cc:654] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:04.796830 14732 tablet_bootstrap.cc:654] T e2257fcb8e1f4a829f1aa7815c588e5a P 79cc4c0a72de4285aabf74e324639cd0: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:04.797283 14731 log.cc:826] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:04.797559 14732 log.cc:826] T e2257fcb8e1f4a829f1aa7815c588e5a P 79cc4c0a72de4285aabf74e324639cd0: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:04.798548 14730 tablet_bootstrap.cc:492] T e2257fcb8e1f4a829f1aa7815c588e5a P 32bba4052eeb44f4ac324e014cd9d426: No bootstrap required, opened a new log
I20260128 15:02:04.798735 14730 ts_tablet_manager.cc:1403] T e2257fcb8e1f4a829f1aa7815c588e5a P 32bba4052eeb44f4ac324e014cd9d426: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260128 15:02:04.798890 14731 tablet_bootstrap.cc:492] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc: No bootstrap required, opened a new log
I20260128 15:02:04.799049 14731 ts_tablet_manager.cc:1403] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260128 15:02:04.799152 14732 tablet_bootstrap.cc:492] T e2257fcb8e1f4a829f1aa7815c588e5a P 79cc4c0a72de4285aabf74e324639cd0: No bootstrap required, opened a new log
I20260128 15:02:04.799309 14732 ts_tablet_manager.cc:1403] T e2257fcb8e1f4a829f1aa7815c588e5a P 79cc4c0a72de4285aabf74e324639cd0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260128 15:02:04.801669 14731 raft_consensus.cc:359] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79cc4c0a72de4285aabf74e324639cd0" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44165 } } peers { permanent_uuid: "32bba4052eeb44f4ac324e014cd9d426" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 40605 } } peers { permanent_uuid: "d5d84f5fc4c34fe0b8feb595f74267cc" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35585 } }
I20260128 15:02:04.801838 14731 raft_consensus.cc:385] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:04.801913 14731 raft_consensus.cc:740] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5d84f5fc4c34fe0b8feb595f74267cc, State: Initialized, Role: FOLLOWER
I20260128 15:02:04.802196 14730 raft_consensus.cc:359] T e2257fcb8e1f4a829f1aa7815c588e5a P 32bba4052eeb44f4ac324e014cd9d426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79cc4c0a72de4285aabf74e324639cd0" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44165 } } peers { permanent_uuid: "32bba4052eeb44f4ac324e014cd9d426" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 40605 } } peers { permanent_uuid: "d5d84f5fc4c34fe0b8feb595f74267cc" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35585 } }
I20260128 15:02:04.802306 14731 consensus_queue.cc:260] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79cc4c0a72de4285aabf74e324639cd0" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44165 } } peers { permanent_uuid: "32bba4052eeb44f4ac324e014cd9d426" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 40605 } } peers { permanent_uuid: "d5d84f5fc4c34fe0b8feb595f74267cc" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35585 } }
I20260128 15:02:04.802515 14730 raft_consensus.cc:385] T e2257fcb8e1f4a829f1aa7815c588e5a P 32bba4052eeb44f4ac324e014cd9d426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:04.802618 14730 raft_consensus.cc:740] T e2257fcb8e1f4a829f1aa7815c588e5a P 32bba4052eeb44f4ac324e014cd9d426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32bba4052eeb44f4ac324e014cd9d426, State: Initialized, Role: FOLLOWER
I20260128 15:02:04.802577 14732 raft_consensus.cc:359] T e2257fcb8e1f4a829f1aa7815c588e5a P 79cc4c0a72de4285aabf74e324639cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79cc4c0a72de4285aabf74e324639cd0" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44165 } } peers { permanent_uuid: "32bba4052eeb44f4ac324e014cd9d426" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 40605 } } peers { permanent_uuid: "d5d84f5fc4c34fe0b8feb595f74267cc" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35585 } }
I20260128 15:02:04.802798 14732 raft_consensus.cc:385] T e2257fcb8e1f4a829f1aa7815c588e5a P 79cc4c0a72de4285aabf74e324639cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:04.802879 14732 raft_consensus.cc:740] T e2257fcb8e1f4a829f1aa7815c588e5a P 79cc4c0a72de4285aabf74e324639cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cc4c0a72de4285aabf74e324639cd0, State: Initialized, Role: FOLLOWER
I20260128 15:02:04.803089 14730 consensus_queue.cc:260] T e2257fcb8e1f4a829f1aa7815c588e5a P 32bba4052eeb44f4ac324e014cd9d426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79cc4c0a72de4285aabf74e324639cd0" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44165 } } peers { permanent_uuid: "32bba4052eeb44f4ac324e014cd9d426" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 40605 } } peers { permanent_uuid: "d5d84f5fc4c34fe0b8feb595f74267cc" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35585 } }
I20260128 15:02:04.803180 14731 ts_tablet_manager.cc:1434] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260128 15:02:04.803263 14732 consensus_queue.cc:260] T e2257fcb8e1f4a829f1aa7815c588e5a P 79cc4c0a72de4285aabf74e324639cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79cc4c0a72de4285aabf74e324639cd0" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44165 } } peers { permanent_uuid: "32bba4052eeb44f4ac324e014cd9d426" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 40605 } } peers { permanent_uuid: "d5d84f5fc4c34fe0b8feb595f74267cc" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35585 } }
I20260128 15:02:04.803521 14441 heartbeater.cc:499] Master 127.13.88.190:38449 was elected leader, sending a full tablet report...
I20260128 15:02:04.804054 14732 ts_tablet_manager.cc:1434] T e2257fcb8e1f4a829f1aa7815c588e5a P 79cc4c0a72de4285aabf74e324639cd0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260128 15:02:04.804375 14577 heartbeater.cc:499] Master 127.13.88.190:38449 was elected leader, sending a full tablet report...
I20260128 15:02:04.804790 14730 ts_tablet_manager.cc:1434] T e2257fcb8e1f4a829f1aa7815c588e5a P 32bba4052eeb44f4ac324e014cd9d426: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260128 15:02:04.805310 14713 heartbeater.cc:499] Master 127.13.88.190:38449 was elected leader, sending a full tablet report...
W20260128 15:02:04.828087 14442 tablet.cc:2378] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 15:02:04.901574 14736 raft_consensus.cc:493] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:02:04.901738 14736 raft_consensus.cc:515] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79cc4c0a72de4285aabf74e324639cd0" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44165 } } peers { permanent_uuid: "32bba4052eeb44f4ac324e014cd9d426" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 40605 } } peers { permanent_uuid: "d5d84f5fc4c34fe0b8feb595f74267cc" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35585 } }
I20260128 15:02:04.902752 14736 leader_election.cc:290] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79cc4c0a72de4285aabf74e324639cd0 (127.13.88.130:44165), 32bba4052eeb44f4ac324e014cd9d426 (127.13.88.131:40605)
I20260128 15:02:04.905582 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.902971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:40605 (local address 127.13.88.129:54877)
0128 15:02:04.903288 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:04.903301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:04.903397 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:04.903637 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:04.903640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:04.903655 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:04.903890 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:04.903896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.904597 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.904601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:04.905170 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:04.905177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.905269 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.905298 (+ 29us) client_negotiation.cc:769] Sending connection context
0128 15:02:04.905346 (+ 48us) client_negotiation.cc:240] Negotiation successful
0128 15:02:04.905402 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
I20260128 15:02:04.905588 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.902921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:44165 (local address 127.13.88.129:51473)
0128 15:02:04.903228 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:04.903239 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:04.903331 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:04.903564 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:04.903567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:04.903582 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:04.903847 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:04.903854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.904607 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.904610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:04.905186 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:04.905194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.905302 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.905321 (+ 19us) client_negotiation.cc:769] Sending connection context
0128 15:02:04.905368 (+ 47us) client_negotiation.cc:240] Negotiation successful
0128 15:02:04.905429 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":86,"threads_started":1}
I20260128 15:02:04.906003 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.903053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:54877 (local address 127.13.88.131:40605)
0128 15:02:04.903177 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:04.903181 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:04.903380 (+ 199us) server_negotiation.cc:408] Connection header received
0128 15:02:04.903486 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:04.903489 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:04.903535 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:04.903612 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:04.903997 (+ 385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.904499 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.905289 (+ 790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.905744 (+ 455us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.905783 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:04.905836 (+ 53us) server_negotiation.cc:300] Negotiation successful
0128 15:02:04.905893 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260128 15:02:04.906010 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.903048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:51473 (local address 127.13.88.130:44165)
0128 15:02:04.903182 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:04.903187 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:04.903309 (+ 122us) server_negotiation.cc:408] Connection header received
0128 15:02:04.903430 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:04.903434 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:04.903487 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:04.903541 (+ 54us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:04.903958 (+ 417us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.904501 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.905309 (+ 808us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.905782 (+ 473us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.905819 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:04.905866 (+ 47us) server_negotiation.cc:300] Negotiation successful
0128 15:02:04.905922 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260128 15:02:04.906526 14667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2257fcb8e1f4a829f1aa7815c588e5a" candidate_uuid: "d5d84f5fc4c34fe0b8feb595f74267cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32bba4052eeb44f4ac324e014cd9d426" is_pre_election: true
I20260128 15:02:04.906525 14531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2257fcb8e1f4a829f1aa7815c588e5a" candidate_uuid: "d5d84f5fc4c34fe0b8feb595f74267cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79cc4c0a72de4285aabf74e324639cd0" is_pre_election: true
I20260128 15:02:04.906755 14667 raft_consensus.cc:2468] T e2257fcb8e1f4a829f1aa7815c588e5a P 32bba4052eeb44f4ac324e014cd9d426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5d84f5fc4c34fe0b8feb595f74267cc in term 0.
I20260128 15:02:04.906757 14531 raft_consensus.cc:2468] T e2257fcb8e1f4a829f1aa7815c588e5a P 79cc4c0a72de4285aabf74e324639cd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5d84f5fc4c34fe0b8feb595f74267cc in term 0.
I20260128 15:02:04.907140 14329 leader_election.cc:304] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32bba4052eeb44f4ac324e014cd9d426, d5d84f5fc4c34fe0b8feb595f74267cc; no voters:
I20260128 15:02:04.907364 14736 raft_consensus.cc:2804] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:02:04.907444 14736 raft_consensus.cc:493] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:02:04.907492 14736 raft_consensus.cc:3060] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:04.908339 14736 raft_consensus.cc:515] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79cc4c0a72de4285aabf74e324639cd0" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44165 } } peers { permanent_uuid: "32bba4052eeb44f4ac324e014cd9d426" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 40605 } } peers { permanent_uuid: "d5d84f5fc4c34fe0b8feb595f74267cc" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35585 } }
I20260128 15:02:04.908659 14736 leader_election.cc:290] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [CANDIDATE]: Term 1 election: Requested vote from peers 79cc4c0a72de4285aabf74e324639cd0 (127.13.88.130:44165), 32bba4052eeb44f4ac324e014cd9d426 (127.13.88.131:40605)
I20260128 15:02:04.908951 14531 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2257fcb8e1f4a829f1aa7815c588e5a" candidate_uuid: "d5d84f5fc4c34fe0b8feb595f74267cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79cc4c0a72de4285aabf74e324639cd0"
I20260128 15:02:04.909003 14667 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e2257fcb8e1f4a829f1aa7815c588e5a" candidate_uuid: "d5d84f5fc4c34fe0b8feb595f74267cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32bba4052eeb44f4ac324e014cd9d426"
I20260128 15:02:04.909061 14531 raft_consensus.cc:3060] T e2257fcb8e1f4a829f1aa7815c588e5a P 79cc4c0a72de4285aabf74e324639cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:04.909097 14667 raft_consensus.cc:3060] T e2257fcb8e1f4a829f1aa7815c588e5a P 32bba4052eeb44f4ac324e014cd9d426 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:04.910183 14667 raft_consensus.cc:2468] T e2257fcb8e1f4a829f1aa7815c588e5a P 32bba4052eeb44f4ac324e014cd9d426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5d84f5fc4c34fe0b8feb595f74267cc in term 1.
I20260128 15:02:04.910183 14531 raft_consensus.cc:2468] T e2257fcb8e1f4a829f1aa7815c588e5a P 79cc4c0a72de4285aabf74e324639cd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5d84f5fc4c34fe0b8feb595f74267cc in term 1.
I20260128 15:02:04.910459 14329 leader_election.cc:304] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32bba4052eeb44f4ac324e014cd9d426, d5d84f5fc4c34fe0b8feb595f74267cc; no voters:
I20260128 15:02:04.910620 14736 raft_consensus.cc:2804] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:04.910830 14736 raft_consensus.cc:697] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [term 1 LEADER]: Becoming Leader. State: Replica: d5d84f5fc4c34fe0b8feb595f74267cc, State: Running, Role: LEADER
I20260128 15:02:04.911110 14736 consensus_queue.cc:237] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79cc4c0a72de4285aabf74e324639cd0" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44165 } } peers { permanent_uuid: "32bba4052eeb44f4ac324e014cd9d426" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 40605 } } peers { permanent_uuid: "d5d84f5fc4c34fe0b8feb595f74267cc" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35585 } }
I20260128 15:02:04.914021 14247 catalog_manager.cc:5654] T e2257fcb8e1f4a829f1aa7815c588e5a P d5d84f5fc4c34fe0b8feb595f74267cc reported cstate change: term changed from 0 to 1, leader changed from <none> to d5d84f5fc4c34fe0b8feb595f74267cc (127.13.88.129). New cstate: current_term: 1 leader_uuid: "d5d84f5fc4c34fe0b8feb595f74267cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79cc4c0a72de4285aabf74e324639cd0" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32bba4052eeb44f4ac324e014cd9d426" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 40605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5d84f5fc4c34fe0b8feb595f74267cc" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35585 } health_report { overall_health: HEALTHY } } }
Jan 28 15:02:04 dist-test-slave-m28t krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.129@KRBTEST.COM
I20260128 15:02:04.936713 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.929338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.13.88.129:35585)
0128 15:02:04.929477 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:04.929481 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:04.929622 (+ 141us) server_negotiation.cc:408] Connection header received
0128 15:02:04.929742 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:04.929746 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:04.929792 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:04.929856 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:04.930626 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.931121 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.931742 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.931933 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.933892 (+ 1959us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:04.933915 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:04.933927 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:04.933953 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:04.935393 (+ 1440us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:04.935729 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:04.935734 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:04.935735 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:04.935787 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:04.935930 (+ 143us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:04.935932 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:04.935933 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:04.936120 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:04.936314 (+ 194us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:04.936487 (+ 173us) server_negotiation.cc:300] Negotiation successful
0128 15:02:04.936588 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jan 28 15:02:04 dist-test-slave-m28t krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612524, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 28 15:02:04 dist-test-slave-m28t krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612524, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.88.129@KRBTEST.COM
I20260128 15:02:04.958726 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.951933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.13.88.129:35585)
0128 15:02:04.952077 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:04.952081 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:04.952139 (+ 58us) server_negotiation.cc:408] Connection header received
0128 15:02:04.952258 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:04.952261 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:04.952303 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:04.952360 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:04.953149 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.953609 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.954230 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.954388 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.956271 (+ 1883us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:04.956292 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:04.956294 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:04.956321 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:04.957603 (+ 1282us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:04.957912 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:04.957915 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:04.957916 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:04.957953 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:04.958083 (+ 130us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:04.958085 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:04.958086 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:04.958205 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:04.958293 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:04.958461 (+ 168us) server_negotiation.cc:300] Negotiation successful
0128 15:02:04.958553 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Jan 28 15:02:04 dist-test-slave-m28t krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 28 15:02:04 dist-test-slave-m28t krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.129@KRBTEST.COM
I20260128 15:02:04.979687 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:04.973389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.13.88.129:35585)
0128 15:02:04.973539 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:04.973543 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:04.973588 (+ 45us) server_negotiation.cc:408] Connection header received
0128 15:02:04.973701 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:04.973703 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:04.973746 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:04.973810 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:04.974508 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.974960 (+ 452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:04.975519 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:04.975637 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:04.977523 (+ 1886us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:04.977540 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:04.977543 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:04.977565 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:04.978634 (+ 1069us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:04.978943 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:04.978945 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:04.978946 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:04.978981 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:04.979111 (+ 130us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:04.979113 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:04.979114 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:04.979224 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:04.979326 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:04.979449 (+ 123us) server_negotiation.cc:300] Negotiation successful
0128 15:02:04.979534 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260128 15:02:04.981002 14355 tablet_service.cc:3012] Rejecting scan request for tablet e2257fcb8e1f4a829f1aa7815c588e5a: Uninitialized: clean time has not yet been initialized
I20260128 15:02:04.982538 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 14312
W20260128 15:02:04.987282 14714 tablet.cc:2378] T e2257fcb8e1f4a829f1aa7815c588e5a P 32bba4052eeb44f4ac324e014cd9d426: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 15:02:04.988230 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 14448
I20260128 15:02:04.992980 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 14584
I20260128 15:02:04.997366 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 14216
2026-01-28T15:02:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3049 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:02:05 dist-test-slave-m28t krb5kdc[14755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:02:05 dist-test-slave-m28t krb5kdc[14755](info): set up 2 sockets
Jan 28 15:02:05 dist-test-slave-m28t krb5kdc[14755](info): commencing operation
krb5kdc: starting...
W20260128 15:02:06.836171 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.818s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Jan 28 15:02:06 dist-test-slave-m28t krb5kdc[14755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:02:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:06.957134 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:39421
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:39421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:07.058105 14771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:07.058301 14771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:07.058357 14771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:07.061475 14771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:07.061525 14771 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260128 15:02:07.061543 14771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:07.061578 14771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:07.061609 14771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:07.061637 14771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:02:07.065694 14771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:39421
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:39421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:07.066656 14771 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:07.067454 14771 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:07.072604 14776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:07.072609 14779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:07.072669 14777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:07.072849 14771 server_base.cc:1060] running on GCE node
I20260128 15:02:07.073254 14771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:07.074041 14771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:07.075179 14771 hybrid_clock.cc:648] HybridClock initialized: now 1769612527075168 us; error 24 us; skew 500 ppm
Jan 28 15:02:07 dist-test-slave-m28t krb5kdc[14755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612527, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:07.077515 14771 init.cc:377] Logged in from keytab as kudu/127.13.88.190@KRBTEST.COM (short username kudu)
I20260128 15:02:07.078469 14771 webserver.cc:492] Webserver started at http://127.13.88.190:45087/ using document root <none> and password file <none>
I20260128 15:02:07.078964 14771 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:07.079008 14771 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:07.079180 14771 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:07.080832 14771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "7d43376280c141e6b76088553cb7ef90"
format_stamp: "Formatted at 2026-01-28 15:02:07 on dist-test-slave-m28t"
server_key: "52562b0f54b3142d6044ed3b3d4e07e2"
server_key_iv: "2c90fa4aeca9656be4be90b2156cebe5"
server_key_version: "encryptionkey@0"
I20260128 15:02:07.081266 14771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "7d43376280c141e6b76088553cb7ef90"
format_stamp: "Formatted at 2026-01-28 15:02:07 on dist-test-slave-m28t"
server_key: "52562b0f54b3142d6044ed3b3d4e07e2"
server_key_iv: "2c90fa4aeca9656be4be90b2156cebe5"
server_key_version: "encryptionkey@0"
I20260128 15:02:07.084342 14771 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260128 15:02:07.086331 14786 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:07.087273 14771 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:07.087366 14771 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "7d43376280c141e6b76088553cb7ef90"
format_stamp: "Formatted at 2026-01-28 15:02:07 on dist-test-slave-m28t"
server_key: "52562b0f54b3142d6044ed3b3d4e07e2"
server_key_iv: "2c90fa4aeca9656be4be90b2156cebe5"
server_key_version: "encryptionkey@0"
I20260128 15:02:07.087453 14771 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:07.100672 14771 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:07.107421 14771 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:07.107583 14771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:07.114886 14771 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:39421
I20260128 15:02:07.115067 14848 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:39421 every 8 connection(s)
I20260128 15:02:07.115847 14771 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:07.118479 14849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:07.121521 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 14771
I20260128 15:02:07.121621 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:07.121843 13666 external_mini_cluster.cc:1468] Setting key 787c01257e993e074a6ec71117642dc8
I20260128 15:02:07.123704 14849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90: Bootstrap starting.
I20260128 15:02:07.125962 14849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:07.126715 14849 log.cc:826] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:07.128679 14849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90: No bootstrap required, opened a new log
Jan 28 15:02:07 dist-test-slave-m28t krb5kdc[14755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:07.131130 14849 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d43376280c141e6b76088553cb7ef90" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39421 } }
I20260128 15:02:07.131304 14849 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:07.131364 14849 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d43376280c141e6b76088553cb7ef90, State: Initialized, Role: FOLLOWER
I20260128 15:02:07.131747 14849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d43376280c141e6b76088553cb7ef90" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39421 } }
I20260128 15:02:07.131863 14849 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:07.131919 14849 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:07.132004 14849 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:07.132771 14849 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d43376280c141e6b76088553cb7ef90" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39421 } }
I20260128 15:02:07.133108 14849 leader_election.cc:304] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d43376280c141e6b76088553cb7ef90; no voters:
I20260128 15:02:07.133383 14849 leader_election.cc:290] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:07.133457 14854 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:07.133641 14854 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [term 1 LEADER]: Becoming Leader. State: Replica: 7d43376280c141e6b76088553cb7ef90, State: Running, Role: LEADER
I20260128 15:02:07.133767 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.123001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.13.88.190:39421)
0128 15:02:07.123355 (+ 354us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:07.123365 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:07.123398 (+ 33us) server_negotiation.cc:408] Connection header received
0128 15:02:07.123904 (+ 506us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:07.123922 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:07.124194 (+ 272us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:07.124487 (+ 293us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:07.125434 (+ 947us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.126173 (+ 739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.126987 (+ 814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.127233 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.129401 (+ 2168us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:07.129418 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:07.129427 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:07.129457 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:07.131432 (+ 1975us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:07.131819 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:07.131825 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:07.131828 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:07.131913 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:07.132133 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:07.132137 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:07.132139 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:07.132489 (+ 350us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:07.132692 (+ 203us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:07.132910 (+ 218us) server_negotiation.cc:300] Negotiation successful
0128 15:02:07.133128 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":102,"threads_started":1}
I20260128 15:02:07.133956 14854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d43376280c141e6b76088553cb7ef90" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39421 } }
I20260128 15:02:07.134485 14849 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:07.135667 14856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d43376280c141e6b76088553cb7ef90. Latest consensus state: current_term: 1 leader_uuid: "7d43376280c141e6b76088553cb7ef90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d43376280c141e6b76088553cb7ef90" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39421 } } }
I20260128 15:02:07.135771 14856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:07.136756 14855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d43376280c141e6b76088553cb7ef90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d43376280c141e6b76088553cb7ef90" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39421 } } }
I20260128 15:02:07.137004 14855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90 [sys.catalog]: This master's current role is: LEADER
W20260128 15:02:07.138883 14870 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 15:02:07.138947 14870 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 15:02:07.139024 14863 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:07.139907 14863 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:07.144726 14863 catalog_manager.cc:1357] Generated new cluster ID: 13587c3ebffe4f3eb182c374b0efac1b
I20260128 15:02:07.144784 14863 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:07.153949 14863 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:07.155093 14863 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:07.163455 14863 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90: Generated new TSK 0
I20260128 15:02:07.164196 14863 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:07.208607 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:39421
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:07.309999 14877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:07.310194 14877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:07.310257 14877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:07.313489 14877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:07.313557 14877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:07.313620 14877 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 15:02:07.313689 14877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:02:07.317817 14877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.88.190:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:07.318817 14877 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:07.319619 14877 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:07.325999 14885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:07.326025 14882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:07.326048 14883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:07.326221 14877 server_base.cc:1060] running on GCE node
I20260128 15:02:07.326566 14877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:07.327075 14877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:07.328218 14877 hybrid_clock.cc:648] HybridClock initialized: now 1769612527328207 us; error 24 us; skew 500 ppm
Jan 28 15:02:07 dist-test-slave-m28t krb5kdc[14755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612527, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:07.330646 14877 init.cc:377] Logged in from keytab as kudu/127.13.88.129@KRBTEST.COM (short username kudu)
I20260128 15:02:07.331631 14877 webserver.cc:492] Webserver started at http://127.13.88.129:37349/ using document root <none> and password file <none>
I20260128 15:02:07.332154 14877 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:07.332202 14877 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:07.332370 14877 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:07.334081 14877 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "11fc6468b6874524948d3ac79522c520"
format_stamp: "Formatted at 2026-01-28 15:02:07 on dist-test-slave-m28t"
server_key: "a946ffc7576a6176d2bd4698de969570"
server_key_iv: "66cce81b1d032ccbe538e61c950db179"
server_key_version: "encryptionkey@0"
I20260128 15:02:07.334529 14877 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "11fc6468b6874524948d3ac79522c520"
format_stamp: "Formatted at 2026-01-28 15:02:07 on dist-test-slave-m28t"
server_key: "a946ffc7576a6176d2bd4698de969570"
server_key_iv: "66cce81b1d032ccbe538e61c950db179"
server_key_version: "encryptionkey@0"
I20260128 15:02:07.337880 14877 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260128 15:02:07.339928 14892 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:07.340942 14877 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:07.341040 14877 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "11fc6468b6874524948d3ac79522c520"
format_stamp: "Formatted at 2026-01-28 15:02:07 on dist-test-slave-m28t"
server_key: "a946ffc7576a6176d2bd4698de969570"
server_key_iv: "66cce81b1d032ccbe538e61c950db179"
server_key_version: "encryptionkey@0"
I20260128 15:02:07.341127 14877 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:07.355775 14877 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:07.358716 14877 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:07.358899 14877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:07.360370 14877 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:07.360453 14877 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:07.360519 14877 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:07.360549 14877 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 28 15:02:07 dist-test-slave-m28t krb5kdc[14755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612527, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:07.371882 14877 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:36063
I20260128 15:02:07.371990 15007 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:36063 every 8 connection(s)
I20260128 15:02:07.372776 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.360946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:54657 (local address 127.13.88.190:39421)
0128 15:02:07.361093 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:07.361097 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:07.361954 (+ 857us) server_negotiation.cc:408] Connection header received
0128 15:02:07.363076 (+ 1122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:07.363081 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:07.363139 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:07.363218 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:07.365021 (+ 1803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.365700 (+ 679us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.366369 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.366556 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.369098 (+ 2542us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:07.369120 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:07.369122 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:07.369152 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:07.370754 (+ 1602us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:07.371242 (+ 488us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:07.371246 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:07.371248 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:07.371302 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:07.371563 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:07.371566 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:07.371568 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:07.371780 (+ 212us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:07.371869 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:07.372448 (+ 579us) server_negotiation.cc:300] Negotiation successful
0128 15:02:07.372592 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260128 15:02:07.373155 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 14877
I20260128 15:02:07.373241 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:02:07.373474 13666 external_mini_cluster.cc:1468] Setting key 836cd5ed7d404b5cf8976cb2f4bcbf5a
I20260128 15:02:07.373492 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.361269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39421 (local address 127.13.88.129:54657)
0128 15:02:07.361783 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:07.361834 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:07.362824 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:07.363420 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:07.363441 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:07.364018 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:07.364865 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.364884 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.365820 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.365824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:07.366253 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.366263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.366498 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.367182 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:07.367220 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:07.368938 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:07.370891 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:07.370897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:07.370910 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:07.371142 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:07.371385 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:07.371388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:07.371390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:07.371479 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:07.371877 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:07.371884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:07.372173 (+ 289us) client_negotiation.cc:769] Sending connection context
0128 15:02:07.372358 (+ 185us) client_negotiation.cc:240] Negotiation successful
0128 15:02:07.372678 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":123,"threads_started":1}
I20260128 15:02:07.372957 14877 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:02:07.375293 15008 heartbeater.cc:344] Connected to a master server at 127.13.88.190:39421
I20260128 15:02:07.376752 15008 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:07.377255 15008 heartbeater.cc:507] Master 127.13.88.190:39421 requested a full tablet report, sending...
I20260128 15:02:07.378571 14802 ts_manager.cc:194] Registered new tserver with Master: 11fc6468b6874524948d3ac79522c520 (127.13.88.129:36063)
I20260128 15:02:07.379707 14802 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.129@KRBTEST.COM'} at 127.13.88.129:54657
WARNING: no policy specified for kudu/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.130@KRBTEST.COM" created.
I20260128 15:02:07.388350 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.376929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:39415 (local address 127.13.88.190:39421)
0128 15:02:07.377045 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:07.377049 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:07.382845 (+ 5796us) server_negotiation.cc:408] Connection header received
0128 15:02:07.382943 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:07.382946 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:07.382989 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:07.383054 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:07.383560 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.384070 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.384782 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.384918 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.385688 (+ 770us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:07.385708 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:07.385711 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:07.385737 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:07.386996 (+ 1259us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:07.387393 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:07.387396 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:07.387398 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:07.387453 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:07.387664 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:07.387667 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:07.387669 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:07.387818 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:07.387891 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:07.388132 (+ 241us) server_negotiation.cc:300] Negotiation successful
0128 15:02:07.388225 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260128 15:02:07.388407 15015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.376895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39421 (local address 127.13.88.129:39415)
0128 15:02:07.382758 (+ 5863us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:07.382772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:07.382871 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:07.383080 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:07.383082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:07.383227 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:07.383467 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.383474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.384156 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.384159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:07.384633 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.384644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.384791 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.385280 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:07.385293 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:07.385572 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:07.387101 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:07.387105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:07.387107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:07.387304 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:07.387526 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:07.387529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:07.387531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:07.387588 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:07.387901 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:07.387904 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:07.387994 (+ 90us) client_negotiation.cc:769] Sending connection context
0128 15:02:07.388099 (+ 105us) client_negotiation.cc:240] Negotiation successful
0128 15:02:07.388225 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5783,"thread_start_us":79,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:07.415001 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:39421
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:07.517781 15019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:07.517983 15019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:07.518038 15019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:07.521241 15019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:07.521298 15019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:07.521349 15019 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 15:02:07.521394 15019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:02:07.525398 15019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.88.190:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:07.526338 15019 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:07.527110 15019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:07.533221 15025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:07.533223 15024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:07.533283 15027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:07.533481 15019 server_base.cc:1060] running on GCE node
I20260128 15:02:07.533819 15019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:07.534298 15019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:07.535439 15019 hybrid_clock.cc:648] HybridClock initialized: now 1769612527535427 us; error 33 us; skew 500 ppm
Jan 28 15:02:07 dist-test-slave-m28t krb5kdc[14755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612527, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:07.537815 15019 init.cc:377] Logged in from keytab as kudu/127.13.88.130@KRBTEST.COM (short username kudu)
I20260128 15:02:07.538751 15019 webserver.cc:492] Webserver started at http://127.13.88.130:36671/ using document root <none> and password file <none>
I20260128 15:02:07.539237 15019 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:07.539283 15019 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:07.539448 15019 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:07.541090 15019 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "1838f8371da247fda657328d3aa798ba"
format_stamp: "Formatted at 2026-01-28 15:02:07 on dist-test-slave-m28t"
server_key: "01ec53d15f7ceebe04a64b2e577488a8"
server_key_iv: "f16a58790758227b1d10a1691e9da98c"
server_key_version: "encryptionkey@0"
I20260128 15:02:07.541517 15019 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "1838f8371da247fda657328d3aa798ba"
format_stamp: "Formatted at 2026-01-28 15:02:07 on dist-test-slave-m28t"
server_key: "01ec53d15f7ceebe04a64b2e577488a8"
server_key_iv: "f16a58790758227b1d10a1691e9da98c"
server_key_version: "encryptionkey@0"
I20260128 15:02:07.544574 15019 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260128 15:02:07.546607 15034 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:07.547531 15019 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:07.547626 15019 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "1838f8371da247fda657328d3aa798ba"
format_stamp: "Formatted at 2026-01-28 15:02:07 on dist-test-slave-m28t"
server_key: "01ec53d15f7ceebe04a64b2e577488a8"
server_key_iv: "f16a58790758227b1d10a1691e9da98c"
server_key_version: "encryptionkey@0"
I20260128 15:02:07.547710 15019 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:07.562009 15019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:07.564242 15019 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:07.564394 15019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:07.565726 15019 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:07.565773 15019 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:07.565821 15019 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:07.565856 15019 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 28 15:02:07 dist-test-slave-m28t krb5kdc[14755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612527, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:07.576010 15019 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:44141
I20260128 15:02:07.576035 15149 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:44141 every 8 connection(s)
I20260128 15:02:07.577111 15019 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
I20260128 15:02:07.577577 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.566309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:40399 (local address 127.13.88.190:39421)
0128 15:02:07.566443 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:07.566448 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:07.567203 (+ 755us) server_negotiation.cc:408] Connection header received
0128 15:02:07.568050 (+ 847us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:07.568054 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:07.568115 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:07.568195 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:07.569852 (+ 1657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.570619 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.571475 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.571677 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.574201 (+ 2524us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:07.574224 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:07.574228 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:07.574259 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:07.575686 (+ 1427us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:07.576169 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:07.576173 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:07.576175 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:07.576231 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:07.576524 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:07.576527 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:07.576530 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:07.576718 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:07.576818 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:07.577278 (+ 460us) server_negotiation.cc:300] Negotiation successful
0128 15:02:07.577410 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260128 15:02:07.578145 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.566626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39421 (local address 127.13.88.130:40399)
0128 15:02:07.567072 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:07.567116 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:07.567858 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:07.568395 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:07.568406 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:07.568942 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:07.569673 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.569690 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.570744 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.570748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:07.571340 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.571352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.571588 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.572151 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:07.572185 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:07.574059 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:07.575822 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:07.575827 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:07.575838 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:07.576055 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:07.576314 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:07.576316 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:07.576317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:07.576439 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:07.576815 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:07.576820 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:07.577030 (+ 210us) client_negotiation.cc:769] Sending connection context
0128 15:02:07.577197 (+ 167us) client_negotiation.cc:240] Negotiation successful
0128 15:02:07.577417 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":122,"threads_started":1}
I20260128 15:02:07.579149 15150 heartbeater.cc:344] Connected to a master server at 127.13.88.190:39421
I20260128 15:02:07.579298 15150 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:07.579610 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 15019
I20260128 15:02:07.579722 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:02:07.579870 15150 heartbeater.cc:507] Master 127.13.88.190:39421 requested a full tablet report, sending...
I20260128 15:02:07.579942 13666 external_mini_cluster.cc:1468] Setting key 2bc679fb7556c4942e8c61047d5ea282
I20260128 15:02:07.580920 14803 ts_manager.cc:194] Registered new tserver with Master: 1838f8371da247fda657328d3aa798ba (127.13.88.130:44141)
I20260128 15:02:07.581641 14803 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.130@KRBTEST.COM'} at 127.13.88.130:40399
I20260128 15:02:07.587461 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.581356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39421 (local address 127.13.88.130:52823)
0128 15:02:07.581600 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:07.581612 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:07.581698 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:07.581929 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:07.581932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:07.582084 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:07.582335 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.582344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.583236 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.583240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:07.583773 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.583784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.583887 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.584354 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:07.584369 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:07.584689 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:07.586324 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:07.586326 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:07.586328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:07.586481 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:07.586701 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:07.586703 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:07.586705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:07.586740 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:07.587106 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:07.587108 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:07.587169 (+ 61us) client_negotiation.cc:769] Sending connection context
0128 15:02:07.587238 (+ 69us) client_negotiation.cc:240] Negotiation successful
0128 15:02:07.587329 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
I20260128 15:02:07.587572 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.581292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:52823 (local address 127.13.88.190:39421)
0128 15:02:07.581398 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:07.581402 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:07.581681 (+ 279us) server_negotiation.cc:408] Connection header received
0128 15:02:07.581772 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:07.581776 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:07.581820 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:07.581890 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:07.582457 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.583121 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.583890 (+ 769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.584028 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.584801 (+ 773us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:07.584818 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:07.584821 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:07.584846 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:07.586223 (+ 1377us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:07.586566 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:07.586570 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:07.586572 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:07.586623 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:07.586810 (+ 187us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:07.586813 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:07.586815 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:07.587026 (+ 211us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:07.587129 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:07.587305 (+ 176us) server_negotiation.cc:300] Negotiation successful
0128 15:02:07.587426 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
WARNING: no policy specified for kudu/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:07.620456 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:39421
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:07.721120 15161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:07.721318 15161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:07.721372 15161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:07.724506 15161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:07.724558 15161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:07.724606 15161 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 15:02:07.724649 15161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:02:07.728686 15161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.88.190:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:07.729638 15161 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:07.730412 15161 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:07.736778 15166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:07.736825 15167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:07.736835 15169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:07.736974 15161 server_base.cc:1060] running on GCE node
I20260128 15:02:07.737376 15161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:07.737911 15161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:07.739054 15161 hybrid_clock.cc:648] HybridClock initialized: now 1769612527739040 us; error 28 us; skew 500 ppm
Jan 28 15:02:07 dist-test-slave-m28t krb5kdc[14755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612527, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:07.741508 15161 init.cc:377] Logged in from keytab as kudu/127.13.88.131@KRBTEST.COM (short username kudu)
I20260128 15:02:07.742493 15161 webserver.cc:492] Webserver started at http://127.13.88.131:37683/ using document root <none> and password file <none>
I20260128 15:02:07.743036 15161 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:07.743083 15161 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:07.743252 15161 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:07.744983 15161 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "e39bde2fce674c3d9d9a040c0663bd24"
format_stamp: "Formatted at 2026-01-28 15:02:07 on dist-test-slave-m28t"
server_key: "5f9b78e1f94f10b5f50315eecf1382a4"
server_key_iv: "3b44bcaecaa314243d0712a7f8598176"
server_key_version: "encryptionkey@0"
I20260128 15:02:07.745430 15161 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "e39bde2fce674c3d9d9a040c0663bd24"
format_stamp: "Formatted at 2026-01-28 15:02:07 on dist-test-slave-m28t"
server_key: "5f9b78e1f94f10b5f50315eecf1382a4"
server_key_iv: "3b44bcaecaa314243d0712a7f8598176"
server_key_version: "encryptionkey@0"
I20260128 15:02:07.748557 15161 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260128 15:02:07.750566 15176 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:07.751525 15161 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:07.751617 15161 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "e39bde2fce674c3d9d9a040c0663bd24"
format_stamp: "Formatted at 2026-01-28 15:02:07 on dist-test-slave-m28t"
server_key: "5f9b78e1f94f10b5f50315eecf1382a4"
server_key_iv: "3b44bcaecaa314243d0712a7f8598176"
server_key_version: "encryptionkey@0"
I20260128 15:02:07.751703 15161 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:07.764719 15161 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:07.767386 15161 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:07.767551 15161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:07.769012 15161 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:07.769063 15161 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:07.769114 15161 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:07.769140 15161 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 28 15:02:07 dist-test-slave-m28t krb5kdc[14755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612527, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:07.779129 15161 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:36271
I20260128 15:02:07.779158 15291 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:36271 every 8 connection(s)
I20260128 15:02:07.780211 15161 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
I20260128 15:02:07.780784 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.769649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:46183 (local address 127.13.88.190:39421)
0128 15:02:07.769793 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:07.769799 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:07.770506 (+ 707us) server_negotiation.cc:408] Connection header received
0128 15:02:07.771415 (+ 909us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:07.771419 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:07.771489 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:07.771576 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:07.773176 (+ 1600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.773946 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.774787 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.774987 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.777383 (+ 2396us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:07.777406 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:07.777410 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:07.777449 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:07.778946 (+ 1497us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:07.779422 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:07.779425 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:07.779434 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:07.779489 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:07.779764 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:07.779767 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:07.779769 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:07.779949 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:07.780044 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:07.780548 (+ 504us) server_negotiation.cc:300] Negotiation successful
0128 15:02:07.780666 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260128 15:02:07.781476 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.769923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39421 (local address 127.13.88.131:46183)
0128 15:02:07.770354 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:07.770396 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:07.771178 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:07.771774 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:07.771785 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:07.772270 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:07.773014 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.773031 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.774070 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.774074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:07.774678 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.774688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.774896 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.775469 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:07.775499 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:07.777257 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:07.779050 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:07.779055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:07.779066 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:07.779320 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:07.779571 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:07.779574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:07.779575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:07.779668 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:07.780060 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:07.780065 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:07.780278 (+ 213us) client_negotiation.cc:769] Sending connection context
0128 15:02:07.780497 (+ 219us) client_negotiation.cc:240] Negotiation successful
0128 15:02:07.780749 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":132,"threads_started":1}
I20260128 15:02:07.782439 15292 heartbeater.cc:344] Connected to a master server at 127.13.88.190:39421
I20260128 15:02:07.782588 15292 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:07.783136 15292 heartbeater.cc:507] Master 127.13.88.190:39421 requested a full tablet report, sending...
I20260128 15:02:07.784124 14802 ts_manager.cc:194] Registered new tserver with Master: e39bde2fce674c3d9d9a040c0663bd24 (127.13.88.131:36271)
I20260128 15:02:07.784806 14802 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.131@KRBTEST.COM'} at 127.13.88.131:46183
I20260128 15:02:07.784842 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 15161
I20260128 15:02:07.784935 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:02:07.785183 13666 external_mini_cluster.cc:1468] Setting key 75b152cbd3653a9fdf293fc4e539a88e
I20260128 15:02:07.787088 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260128 15:02:07.790817 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.784471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39421 (local address 127.13.88.131:54115)
0128 15:02:07.785158 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:07.785172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:07.785264 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:07.785614 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:07.785617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:07.785775 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:07.786043 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.786050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.786919 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.786923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:07.787345 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.787354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.787436 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.787853 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:07.787865 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:07.788121 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:07.789676 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:07.789679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:07.789681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:07.789836 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:07.790035 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:07.790039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:07.790041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:07.790082 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:07.790363 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:07.790366 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:07.790446 (+ 80us) client_negotiation.cc:769] Sending connection context
0128 15:02:07.790530 (+ 84us) client_negotiation.cc:240] Negotiation successful
0128 15:02:07.790647 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":80,"threads_started":1}
I20260128 15:02:07.790813 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.784522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:54115 (local address 127.13.88.190:39421)
0128 15:02:07.784625 (+ 103us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:07.784628 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:07.785241 (+ 613us) server_negotiation.cc:408] Connection header received
0128 15:02:07.785391 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:07.785394 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:07.785440 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:07.785509 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:07.786144 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.786791 (+ 647us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.787461 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.787640 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.788222 (+ 582us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:07.788238 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:07.788241 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:07.788264 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:07.789575 (+ 1311us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:07.789919 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:07.789923 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:07.789924 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:07.789963 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:07.790151 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:07.790155 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:07.790157 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:07.790282 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:07.790363 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:07.790589 (+ 226us) server_negotiation.cc:300] Negotiation successful
0128 15:02:07.790690 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Jan 28 15:02:07 dist-test-slave-m28t krb5kdc[14755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612527, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 28 15:02:07 dist-test-slave-m28t krb5kdc[14755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612527, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:07.809077 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.802568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.13.88.190:39421)
0128 15:02:07.802702 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:07.802705 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:07.802717 (+ 12us) server_negotiation.cc:408] Connection header received
0128 15:02:07.802753 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:07.802755 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:07.802795 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:07.802860 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:07.803625 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.804106 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.804762 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.804949 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.806809 (+ 1860us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:07.806827 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:07.806829 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:07.806853 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:07.807928 (+ 1075us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:07.808244 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:07.808247 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:07.808248 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:07.808285 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:07.808454 (+ 169us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:07.808457 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:07.808458 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:07.808615 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:07.808719 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:07.808875 (+ 156us) server_negotiation.cc:300] Negotiation successful
0128 15:02:07.808960 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260128 15:02:07.811350 14802 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 15:02:07.813705 14802 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:02:07.825709 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.821410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:36063 (local address 127.0.0.1:35214)
0128 15:02:07.822158 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:07.822196 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:07.822355 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:07.823054 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:07.823060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:07.823091 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:07.823388 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.823395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.824505 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.824509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:07.825244 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.825253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.825353 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.825398 (+ 45us) client_negotiation.cc:769] Sending connection context
0128 15:02:07.825491 (+ 93us) client_negotiation.cc:240] Negotiation successful
0128 15:02:07.825570 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"spinlock_wait_cycles":7168,"thread_start_us":88,"threads_started":1}
I20260128 15:02:07.826035 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.821529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:36271 (local address 127.0.0.1:44010)
0128 15:02:07.822157 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:07.822196 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:07.822342 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:07.823054 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:07.823060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:07.823096 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:07.823366 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.823373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.824887 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.824891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:07.825703 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.825712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.825808 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.825826 (+ 18us) client_negotiation.cc:769] Sending connection context
0128 15:02:07.825872 (+ 46us) client_negotiation.cc:240] Negotiation successful
0128 15:02:07.825923 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"spinlock_wait_cycles":13952,"thread_start_us":96,"threads_started":1}
I20260128 15:02:07.826205 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.821818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:44141 (local address 127.0.0.1:54970)
0128 15:02:07.822576 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:07.822588 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:07.822670 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:07.823228 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:07.823231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:07.823247 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:07.823459 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.823466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.824917 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.824920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:07.825710 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:07.825720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.825960 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.825978 (+ 18us) client_negotiation.cc:769] Sending connection context
0128 15:02:07.826021 (+ 43us) client_negotiation.cc:240] Negotiation successful
0128 15:02:07.826079 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":87,"threads_started":1}
I20260128 15:02:07.827005 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.821506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.13.88.129:36063)
0128 15:02:07.822609 (+ 1103us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:07.822616 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:07.822637 (+ 21us) server_negotiation.cc:408] Connection header received
0128 15:02:07.822726 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:07.822731 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:07.822898 (+ 167us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:07.823530 (+ 632us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:07.823665 (+ 135us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.824366 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.826056 (+ 1690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.826596 (+ 540us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.826721 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:07.826784 (+ 63us) server_negotiation.cc:300] Negotiation successful
0128 15:02:07.826874 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1002,"thread_start_us":100,"threads_started":1}
I20260128 15:02:07.827096 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.821990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.13.88.130:44141)
0128 15:02:07.822315 (+ 325us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:07.822322 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:07.822756 (+ 434us) server_negotiation.cc:408] Connection header received
0128 15:02:07.822835 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:07.822839 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:07.822962 (+ 123us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:07.823077 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:07.823579 (+ 502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.824780 (+ 1201us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.826436 (+ 1656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.826800 (+ 364us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.826879 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:07.826926 (+ 47us) server_negotiation.cc:300] Negotiation successful
0128 15:02:07.826983 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":92,"threads_started":1}
I20260128 15:02:07.827122 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:07.822141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.13.88.131:36271)
0128 15:02:07.822594 (+ 453us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:07.822601 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:07.822624 (+ 23us) server_negotiation.cc:408] Connection header received
0128 15:02:07.822702 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:07.822707 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:07.822889 (+ 182us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:07.823490 (+ 601us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:07.823639 (+ 149us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.824773 (+ 1134us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:07.826298 (+ 1525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:07.826786 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:07.826875 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:07.826927 (+ 52us) server_negotiation.cc:300] Negotiation successful
0128 15:02:07.826997 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":101,"threads_started":1}
I20260128 15:02:07.829654 15083 tablet_service.cc:1505] Processing CreateTablet for tablet f3e68019eafd42ad97efe6a170e223af (DEFAULT_TABLE table=test-table [id=805a45340b754782b8d7393ca8a850fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:07.829658 14942 tablet_service.cc:1505] Processing CreateTablet for tablet f3e68019eafd42ad97efe6a170e223af (DEFAULT_TABLE table=test-table [id=805a45340b754782b8d7393ca8a850fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:07.829658 15226 tablet_service.cc:1505] Processing CreateTablet for tablet f3e68019eafd42ad97efe6a170e223af (DEFAULT_TABLE table=test-table [id=805a45340b754782b8d7393ca8a850fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:07.830489 15083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e68019eafd42ad97efe6a170e223af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:07.830503 14942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e68019eafd42ad97efe6a170e223af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:07.830503 15226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e68019eafd42ad97efe6a170e223af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:07.835794 15315 tablet_bootstrap.cc:492] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba: Bootstrap starting.
I20260128 15:02:07.837139 15317 tablet_bootstrap.cc:492] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24: Bootstrap starting.
I20260128 15:02:07.837131 15316 tablet_bootstrap.cc:492] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520: Bootstrap starting.
I20260128 15:02:07.838205 15315 tablet_bootstrap.cc:654] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:07.839183 15317 tablet_bootstrap.cc:654] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:07.839182 15316 tablet_bootstrap.cc:654] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:07.839207 15315 log.cc:826] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:07.840392 15317 log.cc:826] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:07.840392 15316 log.cc:826] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:07.840945 15315 tablet_bootstrap.cc:492] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba: No bootstrap required, opened a new log
I20260128 15:02:07.841152 15315 ts_tablet_manager.cc:1403] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260128 15:02:07.842110 15316 tablet_bootstrap.cc:492] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520: No bootstrap required, opened a new log
I20260128 15:02:07.842108 15317 tablet_bootstrap.cc:492] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24: No bootstrap required, opened a new log
I20260128 15:02:07.842284 15316 ts_tablet_manager.cc:1403] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260128 15:02:07.842284 15317 ts_tablet_manager.cc:1403] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260128 15:02:07.843842 15315 raft_consensus.cc:359] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } }
I20260128 15:02:07.844018 15315 raft_consensus.cc:385] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:07.844097 15315 raft_consensus.cc:740] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1838f8371da247fda657328d3aa798ba, State: Initialized, Role: FOLLOWER
I20260128 15:02:07.844544 15315 consensus_queue.cc:260] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } }
I20260128 15:02:07.845156 15150 heartbeater.cc:499] Master 127.13.88.190:39421 was elected leader, sending a full tablet report...
I20260128 15:02:07.845263 15316 raft_consensus.cc:359] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } }
I20260128 15:02:07.845263 15317 raft_consensus.cc:359] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } }
I20260128 15:02:07.845455 15316 raft_consensus.cc:385] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:07.845450 15317 raft_consensus.cc:385] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:07.845444 15315 ts_tablet_manager.cc:1434] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260128 15:02:07.845501 15317 raft_consensus.cc:740] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e39bde2fce674c3d9d9a040c0663bd24, State: Initialized, Role: FOLLOWER
I20260128 15:02:07.845507 15316 raft_consensus.cc:740] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11fc6468b6874524948d3ac79522c520, State: Initialized, Role: FOLLOWER
I20260128 15:02:07.845916 15317 consensus_queue.cc:260] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } }
I20260128 15:02:07.845916 15316 consensus_queue.cc:260] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } }
I20260128 15:02:07.846736 15317 ts_tablet_manager.cc:1434] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260128 15:02:07.846735 15316 ts_tablet_manager.cc:1434] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260128 15:02:07.846771 15008 heartbeater.cc:499] Master 127.13.88.190:39421 was elected leader, sending a full tablet report...
I20260128 15:02:07.847023 15292 heartbeater.cc:499] Master 127.13.88.190:39421 was elected leader, sending a full tablet report...
W20260128 15:02:07.874902 15009 tablet.cc:2378] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260128 15:02:08.031574 15293 tablet.cc:2378] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260128 15:02:08.078598 15151 tablet.cc:2378] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 15:02:08.135382 15323 raft_consensus.cc:493] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:02:08.135557 15323 raft_consensus.cc:515] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } }
I20260128 15:02:08.136648 15322 raft_consensus.cc:493] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:02:08.136776 15323 leader_election.cc:290] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e39bde2fce674c3d9d9a040c0663bd24 (127.13.88.131:36271), 1838f8371da247fda657328d3aa798ba (127.13.88.130:44141)
I20260128 15:02:08.136817 15322 raft_consensus.cc:515] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } }
I20260128 15:02:08.137995 15322 leader_election.cc:290] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11fc6468b6874524948d3ac79522c520 (127.13.88.129:36063), 1838f8371da247fda657328d3aa798ba (127.13.88.130:44141)
I20260128 15:02:08.141142 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.138145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:36063 (local address 127.13.88.131:41715)
0128 15:02:08.138292 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:08.138305 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:08.138378 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:08.138697 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:08.138700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:08.138718 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:08.138939 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.138945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.139906 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.139910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:08.140792 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.140801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.140897 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.140912 (+ 15us) client_negotiation.cc:769] Sending connection context
0128 15:02:08.140957 (+ 45us) client_negotiation.cc:240] Negotiation successful
0128 15:02:08.141015 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260128 15:02:08.141160 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.137135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:59841 (local address 127.13.88.130:44141)
0128 15:02:08.137386 (+ 251us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:08.137391 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:08.137410 (+ 19us) server_negotiation.cc:408] Connection header received
0128 15:02:08.137603 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:08.137606 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:08.137649 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:08.137717 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:08.138487 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.139170 (+ 683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.140270 (+ 1100us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.140772 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.140811 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:08.141004 (+ 193us) server_negotiation.cc:300] Negotiation successful
0128 15:02:08.141067 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260128 15:02:08.141166 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.137254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:59065 (local address 127.13.88.131:36271)
0128 15:02:08.137513 (+ 259us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:08.137517 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:08.137529 (+ 12us) server_negotiation.cc:408] Connection header received
0128 15:02:08.137576 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:08.137579 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:08.137626 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:08.137721 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:08.138454 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.139145 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.140292 (+ 1147us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.140774 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.140811 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:08.141003 (+ 192us) server_negotiation.cc:300] Negotiation successful
0128 15:02:08.141058 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260128 15:02:08.141458 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.136974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:44141 (local address 127.13.88.129:59841)
0128 15:02:08.137304 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:08.137320 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:08.137443 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:08.138078 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:08.138081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:08.138101 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:08.138353 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.138362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.139289 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.139292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:08.140150 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.140158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.140875 (+ 717us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.140891 (+ 16us) client_negotiation.cc:769] Sending connection context
0128 15:02:08.141256 (+ 365us) client_negotiation.cc:240] Negotiation successful
0128 15:02:08.141327 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":94,"spinlock_wait_cycles":16512,"thread_start_us":57,"threads_started":1}
I20260128 15:02:08.141623 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.138214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:44141 (local address 127.13.88.131:53825)
0128 15:02:08.138764 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:08.138776 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:08.138850 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:08.139225 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:08.139228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:08.139244 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:08.139543 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.139550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.140587 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.140591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:08.141334 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.141343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.141417 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.141430 (+ 13us) client_negotiation.cc:769] Sending connection context
0128 15:02:08.141465 (+ 35us) client_negotiation.cc:240] Negotiation successful
0128 15:02:08.141510 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":105,"threads_started":1}
I20260128 15:02:08.142004 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.136974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:36271 (local address 127.13.88.129:59065)
0128 15:02:08.137304 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:08.137320 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:08.137454 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:08.137741 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:08.137745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:08.137763 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:08.137959 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.137967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.139255 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.139260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:08.140165 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.140175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.140870 (+ 695us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.140886 (+ 16us) client_negotiation.cc:769] Sending connection context
0128 15:02:08.141372 (+ 486us) client_negotiation.cc:240] Negotiation successful
0128 15:02:08.141448 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"spinlock_wait_cycles":42880,"thread_start_us":72,"threads_started":1}
I20260128 15:02:08.142150 15104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3e68019eafd42ad97efe6a170e223af" candidate_uuid: "11fc6468b6874524948d3ac79522c520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1838f8371da247fda657328d3aa798ba" is_pre_election: true
I20260128 15:02:08.142395 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.138361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:41715 (local address 127.13.88.129:36063)
0128 15:02:08.138475 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:08.138479 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:08.138489 (+ 10us) server_negotiation.cc:408] Connection header received
0128 15:02:08.138521 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:08.138523 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:08.138561 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:08.138619 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:08.139054 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.139772 (+ 718us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.141797 (+ 2025us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.142187 (+ 390us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.142218 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:08.142262 (+ 44us) server_negotiation.cc:300] Negotiation successful
0128 15:02:08.142313 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260128 15:02:08.142398 15104 raft_consensus.cc:2468] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11fc6468b6874524948d3ac79522c520 in term 0.
I20260128 15:02:08.142603 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.138503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:53825 (local address 127.13.88.130:44141)
0128 15:02:08.139025 (+ 522us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:08.139029 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:08.139039 (+ 10us) server_negotiation.cc:408] Connection header received
0128 15:02:08.139075 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:08.139078 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:08.139116 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:08.139638 (+ 522us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:08.139732 (+ 94us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.140479 (+ 747us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.141557 (+ 1078us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.142359 (+ 802us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.142389 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:08.142446 (+ 57us) server_negotiation.cc:300] Negotiation successful
0128 15:02:08.142495 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":79,"threads_started":1}
I20260128 15:02:08.142848 14896 leader_election.cc:304] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11fc6468b6874524948d3ac79522c520, 1838f8371da247fda657328d3aa798ba; no voters:
I20260128 15:02:08.142921 15246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3e68019eafd42ad97efe6a170e223af" candidate_uuid: "11fc6468b6874524948d3ac79522c520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39bde2fce674c3d9d9a040c0663bd24" is_pre_election: true
I20260128 15:02:08.143025 14962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3e68019eafd42ad97efe6a170e223af" candidate_uuid: "e39bde2fce674c3d9d9a040c0663bd24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11fc6468b6874524948d3ac79522c520" is_pre_election: true
I20260128 15:02:08.143079 15323 raft_consensus.cc:2804] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:02:08.143086 15246 raft_consensus.cc:2468] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11fc6468b6874524948d3ac79522c520 in term 0.
I20260128 15:02:08.143134 15323 raft_consensus.cc:493] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:02:08.143170 15323 raft_consensus.cc:3060] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:08.143311 15104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3e68019eafd42ad97efe6a170e223af" candidate_uuid: "e39bde2fce674c3d9d9a040c0663bd24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1838f8371da247fda657328d3aa798ba" is_pre_election: true
I20260128 15:02:08.143420 15104 raft_consensus.cc:2468] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e39bde2fce674c3d9d9a040c0663bd24 in term 0.
I20260128 15:02:08.143767 15180 leader_election.cc:304] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1838f8371da247fda657328d3aa798ba, e39bde2fce674c3d9d9a040c0663bd24; no voters:
I20260128 15:02:08.143983 15322 raft_consensus.cc:2804] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:02:08.144045 15322 raft_consensus.cc:493] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:02:08.144093 15322 raft_consensus.cc:3060] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:08.144309 15323 raft_consensus.cc:515] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } }
I20260128 15:02:08.144577 14962 raft_consensus.cc:2393] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e39bde2fce674c3d9d9a040c0663bd24 in current term 1: Already voted for candidate 11fc6468b6874524948d3ac79522c520 in this term.
I20260128 15:02:08.144652 15323 leader_election.cc:290] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [CANDIDATE]: Term 1 election: Requested vote from peers e39bde2fce674c3d9d9a040c0663bd24 (127.13.88.131:36271), 1838f8371da247fda657328d3aa798ba (127.13.88.130:44141)
I20260128 15:02:08.144976 15246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3e68019eafd42ad97efe6a170e223af" candidate_uuid: "11fc6468b6874524948d3ac79522c520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39bde2fce674c3d9d9a040c0663bd24"
I20260128 15:02:08.145187 15104 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3e68019eafd42ad97efe6a170e223af" candidate_uuid: "11fc6468b6874524948d3ac79522c520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1838f8371da247fda657328d3aa798ba"
I20260128 15:02:08.145203 15322 raft_consensus.cc:515] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } }
I20260128 15:02:08.145268 15104 raft_consensus.cc:3060] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:08.145519 15246 raft_consensus.cc:2393] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 11fc6468b6874524948d3ac79522c520 in current term 1: Already voted for candidate e39bde2fce674c3d9d9a040c0663bd24 in this term.
I20260128 15:02:08.145526 15322 leader_election.cc:290] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [CANDIDATE]: Term 1 election: Requested vote from peers 11fc6468b6874524948d3ac79522c520 (127.13.88.129:36063), 1838f8371da247fda657328d3aa798ba (127.13.88.130:44141)
I20260128 15:02:08.145975 14962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3e68019eafd42ad97efe6a170e223af" candidate_uuid: "e39bde2fce674c3d9d9a040c0663bd24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11fc6468b6874524948d3ac79522c520"
I20260128 15:02:08.146096 14962 raft_consensus.cc:2393] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e39bde2fce674c3d9d9a040c0663bd24 in current term 1: Already voted for candidate 11fc6468b6874524948d3ac79522c520 in this term.
I20260128 15:02:08.146291 15104 raft_consensus.cc:2468] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11fc6468b6874524948d3ac79522c520 in term 1.
I20260128 15:02:08.146311 15102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3e68019eafd42ad97efe6a170e223af" candidate_uuid: "e39bde2fce674c3d9d9a040c0663bd24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1838f8371da247fda657328d3aa798ba"
I20260128 15:02:08.146463 15102 raft_consensus.cc:2393] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e39bde2fce674c3d9d9a040c0663bd24 in current term 1: Already voted for candidate 11fc6468b6874524948d3ac79522c520 in this term.
I20260128 15:02:08.146531 14896 leader_election.cc:304] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 11fc6468b6874524948d3ac79522c520, 1838f8371da247fda657328d3aa798ba; no voters: e39bde2fce674c3d9d9a040c0663bd24
I20260128 15:02:08.146665 15323 raft_consensus.cc:2804] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:08.146783 15180 leader_election.cc:304] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e39bde2fce674c3d9d9a040c0663bd24; no voters: 11fc6468b6874524948d3ac79522c520, 1838f8371da247fda657328d3aa798ba
I20260128 15:02:08.146847 15323 raft_consensus.cc:697] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [term 1 LEADER]: Becoming Leader. State: Replica: 11fc6468b6874524948d3ac79522c520, State: Running, Role: LEADER
I20260128 15:02:08.146932 15322 raft_consensus.cc:2749] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260128 15:02:08.147187 15323 consensus_queue.cc:237] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } }
I20260128 15:02:08.149837 14801 catalog_manager.cc:5654] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11fc6468b6874524948d3ac79522c520 (127.13.88.129). New cstate: current_term: 1 leader_uuid: "11fc6468b6874524948d3ac79522c520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } health_report { overall_health: UNKNOWN } } }
I20260128 15:02:08.187680 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.185075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.13.88.129:36063)
0128 15:02:08.185204 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:08.185207 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:08.185220 (+ 13us) server_negotiation.cc:408] Connection header received
0128 15:02:08.185349 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:08.185352 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:08.185407 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:08.185490 (+ 83us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:02:08.185772 (+ 282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.186227 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.186784 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.186903 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.186948 (+ 45us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:02:08.187370 (+ 422us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:02:08.187460 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:08.187500 (+ 40us) server_negotiation.cc:300] Negotiation successful
0128 15:02:08.187546 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260128 15:02:08.192842 15102 raft_consensus.cc:1275] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba [term 1 FOLLOWER]: Refusing update from remote peer 11fc6468b6874524948d3ac79522c520: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:08.193353 15246 raft_consensus.cc:1275] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [term 1 FOLLOWER]: Refusing update from remote peer 11fc6468b6874524948d3ac79522c520: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:08.193351 15323 consensus_queue.cc:1048] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:08.193818 15323 consensus_queue.cc:1048] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:08.209335 14803 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:02:08.209486 14803 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:02:08.211582 14803 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90: Sending DeleteTablet for 3 replicas of tablet f3e68019eafd42ad97efe6a170e223af
I20260128 15:02:08.212215 14942 tablet_service.cc:1552] Processing DeleteTablet for tablet f3e68019eafd42ad97efe6a170e223af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:08 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:35214
I20260128 15:02:08.212215 15226 tablet_service.cc:1552] Processing DeleteTablet for tablet f3e68019eafd42ad97efe6a170e223af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:08 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:44010
I20260128 15:02:08.212738 15340 tablet_replica.cc:333] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24: stopping tablet replica
I20260128 15:02:08.212682 15083 tablet_service.cc:1552] Processing DeleteTablet for tablet f3e68019eafd42ad97efe6a170e223af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:08 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:54970
I20260128 15:02:08.212929 15341 tablet_replica.cc:333] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520: stopping tablet replica
I20260128 15:02:08.213191 15341 raft_consensus.cc:2243] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [term 1 LEADER]: Raft consensus shutting down.
I20260128 15:02:08.213231 15342 tablet_replica.cc:333] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba: stopping tablet replica
I20260128 15:02:08.213223 14803 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260128 15:02:08.213449 15342 raft_consensus.cc:2243] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:02:08.213449 15340 raft_consensus.cc:2243] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260128 15:02:08.213637 14803 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:02:08.213732 15342 raft_consensus.cc:2272] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:08.213732 15340 raft_consensus.cc:2272] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:08.214148 15341 raft_consensus.cc:2272] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:08.215513 15342 ts_tablet_manager.cc:1916] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:08.215728 15341 ts_tablet_manager.cc:1916] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:08.217942 15341 ts_tablet_manager.cc:1929] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:02:08.217932 15342 ts_tablet_manager.cc:1929] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:02:08.218015 15342 log.cc:1199] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/wal/wals/f3e68019eafd42ad97efe6a170e223af
I20260128 15:02:08.218047 15341 log.cc:1199] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/wal/wals/f3e68019eafd42ad97efe6a170e223af
I20260128 15:02:08.218307 15342 ts_tablet_manager.cc:1950] T f3e68019eafd42ad97efe6a170e223af P 1838f8371da247fda657328d3aa798ba: Deleting consensus metadata
I20260128 15:02:08.218307 15341 ts_tablet_manager.cc:1950] T f3e68019eafd42ad97efe6a170e223af P 11fc6468b6874524948d3ac79522c520: Deleting consensus metadata
I20260128 15:02:08.219403 14789 catalog_manager.cc:4985] TS 11fc6468b6874524948d3ac79522c520 (127.13.88.129:36063): tablet f3e68019eafd42ad97efe6a170e223af (table test-table [id=805a45340b754782b8d7393ca8a850fa]) successfully deleted
I20260128 15:02:08.219547 14942 tablet_service.cc:1505] Processing CreateTablet for tablet 25dcbf4ba68b48b0ad24c5ca90643dee (DEFAULT_TABLE table=test-table [id=e1fbd7235f434f9e9edb2990d751ab4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:08.219637 14790 catalog_manager.cc:4985] TS 1838f8371da247fda657328d3aa798ba (127.13.88.130:44141): tablet f3e68019eafd42ad97efe6a170e223af (table test-table [id=805a45340b754782b8d7393ca8a850fa]) successfully deleted
I20260128 15:02:08.219777 14942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25dcbf4ba68b48b0ad24c5ca90643dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:08.220396 15340 ts_tablet_manager.cc:1916] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:08.220659 15226 tablet_service.cc:1505] Processing CreateTablet for tablet 25dcbf4ba68b48b0ad24c5ca90643dee (DEFAULT_TABLE table=test-table [id=e1fbd7235f434f9e9edb2990d751ab4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:08.220897 15226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25dcbf4ba68b48b0ad24c5ca90643dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:08.221818 15316 tablet_bootstrap.cc:492] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520: Bootstrap starting.
I20260128 15:02:08.222786 15316 tablet_bootstrap.cc:654] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:08.222954 15317 tablet_bootstrap.cc:492] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24: Bootstrap starting.
I20260128 15:02:08.223001 15340 ts_tablet_manager.cc:1929] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:02:08.223073 15340 log.cc:1199] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/wal/wals/f3e68019eafd42ad97efe6a170e223af
I20260128 15:02:08.223366 15340 ts_tablet_manager.cc:1950] T f3e68019eafd42ad97efe6a170e223af P e39bde2fce674c3d9d9a040c0663bd24: Deleting consensus metadata
I20260128 15:02:08.223369 15083 tablet_service.cc:1505] Processing CreateTablet for tablet 25dcbf4ba68b48b0ad24c5ca90643dee (DEFAULT_TABLE table=test-table [id=e1fbd7235f434f9e9edb2990d751ab4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:08.223634 15083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25dcbf4ba68b48b0ad24c5ca90643dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:08.223896 15317 tablet_bootstrap.cc:654] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:08.223991 15316 tablet_bootstrap.cc:492] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520: No bootstrap required, opened a new log
I20260128 15:02:08.224062 15316 ts_tablet_manager.cc:1403] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:08.224498 15316 raft_consensus.cc:359] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.224598 15316 raft_consensus.cc:385] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:08.224625 15316 raft_consensus.cc:740] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11fc6468b6874524948d3ac79522c520, State: Initialized, Role: FOLLOWER
I20260128 15:02:08.224731 15316 consensus_queue.cc:260] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.224972 15316 ts_tablet_manager.cc:1434] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:08.225538 15317 tablet_bootstrap.cc:492] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24: No bootstrap required, opened a new log
I20260128 15:02:08.225589 15315 tablet_bootstrap.cc:492] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba: Bootstrap starting.
I20260128 15:02:08.225616 15317 ts_tablet_manager.cc:1403] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 15:02:08.225965 14787 catalog_manager.cc:4985] TS e39bde2fce674c3d9d9a040c0663bd24 (127.13.88.131:36271): tablet f3e68019eafd42ad97efe6a170e223af (table test-table [id=805a45340b754782b8d7393ca8a850fa]) successfully deleted
I20260128 15:02:08.226027 15317 raft_consensus.cc:359] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.226109 15317 raft_consensus.cc:385] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:08.226137 15317 raft_consensus.cc:740] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e39bde2fce674c3d9d9a040c0663bd24, State: Initialized, Role: FOLLOWER
I20260128 15:02:08.226243 15317 consensus_queue.cc:260] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.226436 15315 tablet_bootstrap.cc:654] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:08.226467 15317 ts_tablet_manager.cc:1434] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:08.227559 15315 tablet_bootstrap.cc:492] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba: No bootstrap required, opened a new log
I20260128 15:02:08.227648 15315 ts_tablet_manager.cc:1403] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260128 15:02:08.228061 15315 raft_consensus.cc:359] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.228144 15315 raft_consensus.cc:385] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:08.228164 15315 raft_consensus.cc:740] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1838f8371da247fda657328d3aa798ba, State: Initialized, Role: FOLLOWER
I20260128 15:02:08.228273 15315 consensus_queue.cc:260] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.228529 15315 ts_tablet_manager.cc:1434] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:08.265462 15323 raft_consensus.cc:493] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:02:08.265597 15323 raft_consensus.cc:515] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.265997 15323 leader_election.cc:290] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1838f8371da247fda657328d3aa798ba (127.13.88.130:44141), e39bde2fce674c3d9d9a040c0663bd24 (127.13.88.131:36271)
I20260128 15:02:08.266383 15246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25dcbf4ba68b48b0ad24c5ca90643dee" candidate_uuid: "11fc6468b6874524948d3ac79522c520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39bde2fce674c3d9d9a040c0663bd24" is_pre_election: true
I20260128 15:02:08.266383 15102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25dcbf4ba68b48b0ad24c5ca90643dee" candidate_uuid: "11fc6468b6874524948d3ac79522c520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1838f8371da247fda657328d3aa798ba" is_pre_election: true
I20260128 15:02:08.266544 15102 raft_consensus.cc:2468] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11fc6468b6874524948d3ac79522c520 in term 0.
I20260128 15:02:08.266544 15246 raft_consensus.cc:2468] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11fc6468b6874524948d3ac79522c520 in term 0.
I20260128 15:02:08.266816 14896 leader_election.cc:304] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11fc6468b6874524948d3ac79522c520, 1838f8371da247fda657328d3aa798ba; no voters:
I20260128 15:02:08.266983 15323 raft_consensus.cc:2804] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:02:08.267058 15323 raft_consensus.cc:493] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:02:08.267103 15323 raft_consensus.cc:3060] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:08.268229 15323 raft_consensus.cc:515] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.268653 15323 leader_election.cc:290] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [CANDIDATE]: Term 1 election: Requested vote from peers 1838f8371da247fda657328d3aa798ba (127.13.88.130:44141), e39bde2fce674c3d9d9a040c0663bd24 (127.13.88.131:36271)
I20260128 15:02:08.268968 15102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25dcbf4ba68b48b0ad24c5ca90643dee" candidate_uuid: "11fc6468b6874524948d3ac79522c520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1838f8371da247fda657328d3aa798ba"
I20260128 15:02:08.269047 15246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "25dcbf4ba68b48b0ad24c5ca90643dee" candidate_uuid: "11fc6468b6874524948d3ac79522c520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39bde2fce674c3d9d9a040c0663bd24"
I20260128 15:02:08.269086 15102 raft_consensus.cc:3060] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:08.269172 15246 raft_consensus.cc:3060] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:08.270002 15246 raft_consensus.cc:2468] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11fc6468b6874524948d3ac79522c520 in term 1.
I20260128 15:02:08.270260 14893 leader_election.cc:304] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11fc6468b6874524948d3ac79522c520, e39bde2fce674c3d9d9a040c0663bd24; no voters:
I20260128 15:02:08.270599 15102 raft_consensus.cc:2468] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11fc6468b6874524948d3ac79522c520 in term 1.
I20260128 15:02:08.270793 15323 raft_consensus.cc:2804] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:08.270879 15323 raft_consensus.cc:697] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [term 1 LEADER]: Becoming Leader. State: Replica: 11fc6468b6874524948d3ac79522c520, State: Running, Role: LEADER
I20260128 15:02:08.271003 15323 consensus_queue.cc:237] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.272629 14803 catalog_manager.cc:5654] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11fc6468b6874524948d3ac79522c520 (127.13.88.129). New cstate: current_term: 1 leader_uuid: "11fc6468b6874524948d3ac79522c520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } health_report { overall_health: UNKNOWN } } }
Jan 28 15:02:08 dist-test-slave-m28t krb5kdc[14755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612527, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:08.317444 15352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.310120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39421 (local address 127.0.0.1:38768)
0128 15:02:08.310449 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:08.310463 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:08.310573 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:08.310801 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:08.310803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:08.311224 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:08.311406 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.311413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.312116 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.312119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:08.312520 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.312529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.312628 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.313220 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:08.313233 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:08.314580 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:08.316098 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:08.316107 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:08.316113 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:08.316320 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:08.316562 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:08.316565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:08.316567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:08.316610 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:08.316912 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:08.316920 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:08.317010 (+ 90us) client_negotiation.cc:769] Sending connection context
0128 15:02:08.317108 (+ 98us) client_negotiation.cc:240] Negotiation successful
0128 15:02:08.317250 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":71,"threads_started":1}
I20260128 15:02:08.317457 15353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.310191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.13.88.190:39421)
0128 15:02:08.310429 (+ 238us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:08.310444 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:08.310529 (+ 85us) server_negotiation.cc:408] Connection header received
0128 15:02:08.310663 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:08.310666 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:08.310703 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:08.310770 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:08.311518 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.312020 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.312618 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.312780 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.314687 (+ 1907us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:08.314702 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:08.314705 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:08.314726 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:08.315997 (+ 1271us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:08.316405 (+ 408us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:08.316408 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:08.316410 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:08.316486 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:08.316680 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:08.316683 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:08.316685 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:08.316818 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:08.316899 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:08.317162 (+ 263us) server_negotiation.cc:300] Negotiation successful
0128 15:02:08.317320 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
I20260128 15:02:08.321115 14803 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:38768:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260128 15:02:08.321768 14803 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:02:08.326778 15226 tablet_service.cc:1505] Processing CreateTablet for tablet 04923bea41da4c198efc3afc0b890216 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4feb5fd1bb534a2da79ec692ea7ae89c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 15:02:08.326761 14942 tablet_service.cc:1505] Processing CreateTablet for tablet 04923bea41da4c198efc3afc0b890216 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4feb5fd1bb534a2da79ec692ea7ae89c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 15:02:08.327108 14942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04923bea41da4c198efc3afc0b890216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:08.327109 15226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04923bea41da4c198efc3afc0b890216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:08.329644 15316 tablet_bootstrap.cc:492] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520: Bootstrap starting.
I20260128 15:02:08.329792 15317 tablet_bootstrap.cc:492] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24: Bootstrap starting.
I20260128 15:02:08.329932 15083 tablet_service.cc:1505] Processing CreateTablet for tablet 04923bea41da4c198efc3afc0b890216 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4feb5fd1bb534a2da79ec692ea7ae89c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 15:02:08.330257 15083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04923bea41da4c198efc3afc0b890216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:08.330575 15316 tablet_bootstrap.cc:654] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:08.330705 15317 tablet_bootstrap.cc:654] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:08.332757 15315 tablet_bootstrap.cc:492] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba: Bootstrap starting.
I20260128 15:02:08.333320 15317 tablet_bootstrap.cc:492] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24: No bootstrap required, opened a new log
I20260128 15:02:08.333402 15317 ts_tablet_manager.cc:1403] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260128 15:02:08.333607 15316 tablet_bootstrap.cc:492] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520: No bootstrap required, opened a new log
I20260128 15:02:08.333680 15316 ts_tablet_manager.cc:1403] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260128 15:02:08.333750 15315 tablet_bootstrap.cc:654] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:08.333858 15317 raft_consensus.cc:359] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.333961 15317 raft_consensus.cc:385] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:08.333988 15317 raft_consensus.cc:740] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e39bde2fce674c3d9d9a040c0663bd24, State: Initialized, Role: FOLLOWER
I20260128 15:02:08.334105 15317 consensus_queue.cc:260] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.334126 15316 raft_consensus.cc:359] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.334231 15316 raft_consensus.cc:385] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:08.334259 15316 raft_consensus.cc:740] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11fc6468b6874524948d3ac79522c520, State: Initialized, Role: FOLLOWER
I20260128 15:02:08.334345 15317 ts_tablet_manager.cc:1434] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:08.334370 15316 consensus_queue.cc:260] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.334523 15322 tablet_replica.cc:442] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } }
I20260128 15:02:08.334625 15322 tablet_replica.cc:445] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:02:08.334635 15316 ts_tablet_manager.cc:1434] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:08.334865 15323 tablet_replica.cc:442] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } }
I20260128 15:02:08.334959 15323 tablet_replica.cc:445] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:02:08.335227 15315 tablet_bootstrap.cc:492] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba: No bootstrap required, opened a new log
I20260128 15:02:08.335296 15315 ts_tablet_manager.cc:1403] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 15:02:08.335702 15315 raft_consensus.cc:359] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.335816 15315 raft_consensus.cc:385] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:08.335850 15315 raft_consensus.cc:740] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1838f8371da247fda657328d3aa798ba, State: Initialized, Role: FOLLOWER
I20260128 15:02:08.335958 15315 consensus_queue.cc:260] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.336189 15315 ts_tablet_manager.cc:1434] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:08.336800 15321 tablet_replica.cc:442] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } }
I20260128 15:02:08.336910 15321 tablet_replica.cc:445] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:02:08.351541 15321 raft_consensus.cc:493] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:02:08.351663 15321 raft_consensus.cc:515] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.352741 15321 leader_election.cc:290] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11fc6468b6874524948d3ac79522c520 (127.13.88.129:36063), e39bde2fce674c3d9d9a040c0663bd24 (127.13.88.131:36271)
I20260128 15:02:08.356101 15358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.352955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:36063 (local address 127.13.88.130:47633)
0128 15:02:08.353566 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:08.353579 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:08.353662 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:08.353910 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:08.353913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:08.353926 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:08.354164 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.354170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.354854 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.354857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:08.355716 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.355725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.355829 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.355851 (+ 22us) client_negotiation.cc:769] Sending connection context
0128 15:02:08.355896 (+ 45us) client_negotiation.cc:240] Negotiation successful
0128 15:02:08.355954 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":338,"thread_start_us":113,"threads_started":1}
I20260128 15:02:08.356683 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.353124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:47633 (local address 127.13.88.129:36063)
0128 15:02:08.353282 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:08.353286 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:08.353655 (+ 369us) server_negotiation.cc:408] Connection header received
0128 15:02:08.353746 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:08.353750 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:08.353805 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:08.353897 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:08.354266 (+ 369us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.354741 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.355861 (+ 1120us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.356374 (+ 513us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.356446 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:08.356499 (+ 53us) server_negotiation.cc:300] Negotiation successful
0128 15:02:08.356569 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260128 15:02:08.356899 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.353072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:49157 (local address 127.13.88.131:36271)
0128 15:02:08.353276 (+ 204us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:08.353281 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:08.353365 (+ 84us) server_negotiation.cc:408] Connection header received
0128 15:02:08.353527 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:08.353531 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:08.353588 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:08.353640 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:08.354200 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.354953 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.355996 (+ 1043us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.356476 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.356512 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:08.356716 (+ 204us) server_negotiation.cc:300] Negotiation successful
0128 15:02:08.356791 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260128 15:02:08.357275 15357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.352954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:36271 (local address 127.13.88.130:49157)
0128 15:02:08.353267 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:08.353282 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:08.353395 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:08.353783 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:08.353785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:08.353799 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:08.354074 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.354081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.355057 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.355061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:08.355875 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.355882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.356568 (+ 686us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.356584 (+ 16us) client_negotiation.cc:769] Sending connection context
0128 15:02:08.357076 (+ 492us) client_negotiation.cc:240] Negotiation successful
0128 15:02:08.357147 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":110,"threads_started":1}
I20260128 15:02:08.357242 14962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04923bea41da4c198efc3afc0b890216" candidate_uuid: "1838f8371da247fda657328d3aa798ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11fc6468b6874524948d3ac79522c520" is_pre_election: true
I20260128 15:02:08.357403 14962 raft_consensus.cc:2468] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1838f8371da247fda657328d3aa798ba in term 0.
I20260128 15:02:08.357748 15246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04923bea41da4c198efc3afc0b890216" candidate_uuid: "1838f8371da247fda657328d3aa798ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39bde2fce674c3d9d9a040c0663bd24" is_pre_election: true
I20260128 15:02:08.357782 15037 leader_election.cc:304] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11fc6468b6874524948d3ac79522c520, 1838f8371da247fda657328d3aa798ba; no voters:
I20260128 15:02:08.357868 15246 raft_consensus.cc:2468] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1838f8371da247fda657328d3aa798ba in term 0.
I20260128 15:02:08.358040 15321 raft_consensus.cc:2804] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:02:08.358103 15321 raft_consensus.cc:493] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:02:08.358140 15321 raft_consensus.cc:3060] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:08.359004 15321 raft_consensus.cc:515] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.359311 15321 leader_election.cc:290] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [CANDIDATE]: Term 1 election: Requested vote from peers 11fc6468b6874524948d3ac79522c520 (127.13.88.129:36063), e39bde2fce674c3d9d9a040c0663bd24 (127.13.88.131:36271)
I20260128 15:02:08.359659 14962 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04923bea41da4c198efc3afc0b890216" candidate_uuid: "1838f8371da247fda657328d3aa798ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11fc6468b6874524948d3ac79522c520"
I20260128 15:02:08.359694 15246 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04923bea41da4c198efc3afc0b890216" candidate_uuid: "1838f8371da247fda657328d3aa798ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39bde2fce674c3d9d9a040c0663bd24"
I20260128 15:02:08.359763 14962 raft_consensus.cc:3060] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:08.359802 15246 raft_consensus.cc:3060] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:08.360926 14962 raft_consensus.cc:2468] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1838f8371da247fda657328d3aa798ba in term 1.
I20260128 15:02:08.360940 15246 raft_consensus.cc:2468] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1838f8371da247fda657328d3aa798ba in term 1.
I20260128 15:02:08.361228 15035 leader_election.cc:304] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1838f8371da247fda657328d3aa798ba, e39bde2fce674c3d9d9a040c0663bd24; no voters:
I20260128 15:02:08.361459 15321 raft_consensus.cc:2804] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:08.361657 15321 raft_consensus.cc:697] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [term 1 LEADER]: Becoming Leader. State: Replica: 1838f8371da247fda657328d3aa798ba, State: Running, Role: LEADER
I20260128 15:02:08.361919 15321 consensus_queue.cc:237] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } }
I20260128 15:02:08.363252 15359 tablet_replica.cc:442] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba: TxnStatusTablet state changed. Reason: New leader 1838f8371da247fda657328d3aa798ba. Latest consensus state: current_term: 1 leader_uuid: "1838f8371da247fda657328d3aa798ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } }
I20260128 15:02:08.363382 15359 tablet_replica.cc:445] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba: This TxnStatusTablet replica's current role is: LEADER
I20260128 15:02:08.363871 15361 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260128 15:02:08.364010 15361 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260128 15:02:08.364365 14803 catalog_manager.cc:5654] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 1838f8371da247fda657328d3aa798ba (127.13.88.130). New cstate: current_term: 1 leader_uuid: "1838f8371da247fda657328d3aa798ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } health_report { overall_health: UNKNOWN } } }
I20260128 15:02:08.386227 15352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.381992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:44141 (local address 127.0.0.1:54980)
0128 15:02:08.382224 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:08.382238 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:08.382363 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:08.382800 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:08.382804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:08.382812 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 15:02:08.383041 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.383071 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.384113 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.384116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:08.384777 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.384787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.384887 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.384930 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 15:02:08.385939 (+ 1009us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 15:02:08.385955 (+ 16us) client_negotiation.cc:769] Sending connection context
0128 15:02:08.386013 (+ 58us) client_negotiation.cc:240] Negotiation successful
0128 15:02:08.386086 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260128 15:02:08.386847 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.382074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54980 (local address 127.13.88.130:44141)
0128 15:02:08.382205 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:08.382210 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:08.382296 (+ 86us) server_negotiation.cc:408] Connection header received
0128 15:02:08.382530 (+ 234us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:08.382534 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:08.382597 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:08.382675 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:02:08.383214 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.383982 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.385039 (+ 1057us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.385235 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.385290 (+ 55us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:02:08.385793 (+ 503us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:02:08.385896 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:08.386663 (+ 767us) server_negotiation.cc:300] Negotiation successful
0128 15:02:08.386728 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260128 15:02:08.390520 15246 raft_consensus.cc:1275] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24 [term 1 FOLLOWER]: Refusing update from remote peer 1838f8371da247fda657328d3aa798ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:08.391068 15359 consensus_queue.cc:1048] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:08.391997 14962 raft_consensus.cc:1275] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520 [term 1 FOLLOWER]: Refusing update from remote peer 1838f8371da247fda657328d3aa798ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:08.392397 15359 consensus_queue.cc:1048] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:08.393748 15322 tablet_replica.cc:442] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24: TxnStatusTablet state changed. Reason: New leader 1838f8371da247fda657328d3aa798ba. Latest consensus state: current_term: 1 leader_uuid: "1838f8371da247fda657328d3aa798ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } }
I20260128 15:02:08.393844 15322 tablet_replica.cc:445] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:02:08.395550 15323 tablet_replica.cc:442] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520: TxnStatusTablet state changed. Reason: New leader 1838f8371da247fda657328d3aa798ba. Latest consensus state: current_term: 1 leader_uuid: "1838f8371da247fda657328d3aa798ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } }
I20260128 15:02:08.395651 15323 tablet_replica.cc:445] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:02:08.397368 15321 tablet_replica.cc:442] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1838f8371da247fda657328d3aa798ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } }
I20260128 15:02:08.397473 15321 tablet_replica.cc:445] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba: This TxnStatusTablet replica's current role is: LEADER
I20260128 15:02:08.397754 15322 tablet_replica.cc:442] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1838f8371da247fda657328d3aa798ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } }
I20260128 15:02:08.397786 15333 mvcc.cc:204] Tried to move back new op lower bound from 7248332916278648832 to 7248332916173586432. Current Snapshot: MvccSnapshot[applied={T|T < 7248332916278648832}]
I20260128 15:02:08.397840 15322 tablet_replica.cc:445] T 04923bea41da4c198efc3afc0b890216 P e39bde2fce674c3d9d9a040c0663bd24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:02:08.398916 15321 tablet_replica.cc:442] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1838f8371da247fda657328d3aa798ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } }
I20260128 15:02:08.399004 15321 tablet_replica.cc:445] T 04923bea41da4c198efc3afc0b890216 P 1838f8371da247fda657328d3aa798ba: This TxnStatusTablet replica's current role is: LEADER
I20260128 15:02:08.399806 15331 mvcc.cc:204] Tried to move back new op lower bound from 7248332916278648832 to 7248332916173586432. Current Snapshot: MvccSnapshot[applied={T|T < 7248332916278648832}]
I20260128 15:02:08.400188 15323 tablet_replica.cc:442] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1838f8371da247fda657328d3aa798ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11fc6468b6874524948d3ac79522c520" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 36063 } } peers { permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 } } peers { permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 } } }
I20260128 15:02:08.400274 15323 tablet_replica.cc:445] T 04923bea41da4c198efc3afc0b890216 P 11fc6468b6874524948d3ac79522c520: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:02:08.410744 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.407984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:44141 (local address 127.13.88.129:36343)
0128 15:02:08.408230 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:08.408242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:08.408328 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:08.408558 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:08.408560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:08.408566 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 15:02:08.408757 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.408762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.409668 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.409670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:08.410065 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.410070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.410146 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.410179 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 15:02:08.410539 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 15:02:08.410542 (+ 3us) client_negotiation.cc:769] Sending connection context
0128 15:02:08.410588 (+ 46us) client_negotiation.cc:240] Negotiation successful
0128 15:02:08.410645 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260128 15:02:08.410776 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.408077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:36343 (local address 127.13.88.130:44141)
0128 15:02:08.408196 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:08.408201 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:08.408305 (+ 104us) server_negotiation.cc:408] Connection header received
0128 15:02:08.408403 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:08.408407 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:08.408478 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:08.408557 (+ 79us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:02:08.408840 (+ 283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.409581 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.410149 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.410338 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.410382 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:02:08.410454 (+ 72us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:02:08.410526 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:08.410611 (+ 85us) server_negotiation.cc:300] Negotiation successful
0128 15:02:08.410652 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260128 15:02:08.416762 15246 raft_consensus.cc:1275] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24 [term 1 FOLLOWER]: Refusing update from remote peer 11fc6468b6874524948d3ac79522c520: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:08.416846 15102 raft_consensus.cc:1275] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba [term 1 FOLLOWER]: Refusing update from remote peer 11fc6468b6874524948d3ac79522c520: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:08.417094 15328 consensus_queue.cc:1048] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e39bde2fce674c3d9d9a040c0663bd24" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:08.417555 15328 consensus_queue.cc:1048] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1838f8371da247fda657328d3aa798ba" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 44141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:08.421072 15332 mvcc.cc:204] Tried to move back new op lower bound from 7248332916392108032 to 7248332915799289856. Current Snapshot: MvccSnapshot[applied={T|T < 7248332916392108032}]
I20260128 15:02:08.438920 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.435868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:33729 (local address 127.13.88.129:36063)
0128 15:02:08.435991 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:08.435996 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:08.436191 (+ 195us) server_negotiation.cc:408] Connection header received
0128 15:02:08.436303 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:08.436307 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:08.436358 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:08.436460 (+ 102us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:02:08.436842 (+ 382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.437407 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.438410 (+ 1003us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.438542 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.438578 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:02:08.438635 (+ 57us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:02:08.438698 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:08.438785 (+ 87us) server_negotiation.cc:300] Negotiation successful
0128 15:02:08.438823 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260128 15:02:08.439007 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.435810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:36063 (local address 127.13.88.130:33729)
0128 15:02:08.436109 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:08.436122 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:08.436210 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:08.436468 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:08.436471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:08.436480 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 15:02:08.436741 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.436748 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.437518 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.437522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:08.438303 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:08.438314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.438426 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.438472 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 15:02:08.438714 (+ 242us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 15:02:08.438719 (+ 5us) client_negotiation.cc:769] Sending connection context
0128 15:02:08.438779 (+ 60us) client_negotiation.cc:240] Negotiation successful
0128 15:02:08.438850 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":107,"threads_started":1}
I20260128 15:02:08.456506 14801 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:02:08.456676 14801 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:02:08.458626 14801 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 7d43376280c141e6b76088553cb7ef90: Sending DeleteTablet for 3 replicas of tablet 25dcbf4ba68b48b0ad24c5ca90643dee
I20260128 15:02:08.459097 14942 tablet_service.cc:1552] Processing DeleteTablet for tablet 25dcbf4ba68b48b0ad24c5ca90643dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:08 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:35214
I20260128 15:02:08.459268 15341 tablet_replica.cc:333] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520: stopping tablet replica
I20260128 15:02:08.459283 15083 tablet_service.cc:1552] Processing DeleteTablet for tablet 25dcbf4ba68b48b0ad24c5ca90643dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:08 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:54970
I20260128 15:02:08.459375 15341 raft_consensus.cc:2243] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [term 1 LEADER]: Raft consensus shutting down.
I20260128 15:02:08.459363 15226 tablet_service.cc:1552] Processing DeleteTablet for tablet 25dcbf4ba68b48b0ad24c5ca90643dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:08 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:44010
I20260128 15:02:08.459467 15342 tablet_replica.cc:333] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba: stopping tablet replica
I20260128 15:02:08.459546 15341 raft_consensus.cc:2272] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:08.459568 15342 raft_consensus.cc:2243] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:02:08.459651 15342 raft_consensus.cc:2272] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:08.459709 15340 tablet_replica.cc:333] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24: stopping tablet replica
I20260128 15:02:08.459800 15340 raft_consensus.cc:2243] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:02:08.459887 15340 raft_consensus.cc:2272] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:08.460700 15340 ts_tablet_manager.cc:1916] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:08.460700 15341 ts_tablet_manager.cc:1916] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:08.460987 15342 ts_tablet_manager.cc:1916] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:08.463150 15340 ts_tablet_manager.cc:1929] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 15:02:08.463150 15341 ts_tablet_manager.cc:1929] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 15:02:08.463212 15340 log.cc:1199] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-2/wal/wals/25dcbf4ba68b48b0ad24c5ca90643dee
I20260128 15:02:08.463212 15341 log.cc:1199] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-0/wal/wals/25dcbf4ba68b48b0ad24c5ca90643dee
I20260128 15:02:08.463393 15342 ts_tablet_manager.cc:1929] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 15:02:08.463455 15341 ts_tablet_manager.cc:1950] T 25dcbf4ba68b48b0ad24c5ca90643dee P 11fc6468b6874524948d3ac79522c520: Deleting consensus metadata
I20260128 15:02:08.463461 15342 log.cc:1199] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1769612519049359-13666-0/minicluster-data/ts-1/wal/wals/25dcbf4ba68b48b0ad24c5ca90643dee
I20260128 15:02:08.463464 15340 ts_tablet_manager.cc:1950] T 25dcbf4ba68b48b0ad24c5ca90643dee P e39bde2fce674c3d9d9a040c0663bd24: Deleting consensus metadata
I20260128 15:02:08.463722 15342 ts_tablet_manager.cc:1950] T 25dcbf4ba68b48b0ad24c5ca90643dee P 1838f8371da247fda657328d3aa798ba: Deleting consensus metadata
I20260128 15:02:08.464025 14787 catalog_manager.cc:4985] TS e39bde2fce674c3d9d9a040c0663bd24 (127.13.88.131:36271): tablet 25dcbf4ba68b48b0ad24c5ca90643dee (table test-table [id=e1fbd7235f434f9e9edb2990d751ab4e]) successfully deleted
I20260128 15:02:08.464049 14789 catalog_manager.cc:4985] TS 11fc6468b6874524948d3ac79522c520 (127.13.88.129:36063): tablet 25dcbf4ba68b48b0ad24c5ca90643dee (table test-table [id=e1fbd7235f434f9e9edb2990d751ab4e]) successfully deleted
I20260128 15:02:08.464644 14790 catalog_manager.cc:4985] TS 1838f8371da247fda657328d3aa798ba (127.13.88.130:44141): tablet 25dcbf4ba68b48b0ad24c5ca90643dee (table test-table [id=e1fbd7235f434f9e9edb2990d751ab4e]) successfully deleted
Jan 28 15:02:08 dist-test-slave-m28t krb5kdc[14755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612527, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.88.129@KRBTEST.COM
I20260128 15:02:08.472112 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.464610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.13.88.129:36063)
0128 15:02:08.464730 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:08.464734 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:08.464949 (+ 215us) server_negotiation.cc:408] Connection header received
0128 15:02:08.465053 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:08.465056 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:08.465100 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:08.465160 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:08.465922 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.466373 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.466974 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.467153 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.469153 (+ 2000us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:08.469172 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:08.469184 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:08.469206 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:08.470706 (+ 1500us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:08.471076 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:08.471081 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:08.471082 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:08.471140 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:08.471335 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:08.471338 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:08.471341 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:08.471518 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:08.471727 (+ 209us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:08.471868 (+ 141us) server_negotiation.cc:300] Negotiation successful
0128 15:02:08.471976 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260128 15:02:08.472556 15002 server_base.cc:1142] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35234
I20260128 15:02:08.479862 15353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:08.474175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.13.88.190:39421)
0128 15:02:08.474320 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:08.474324 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:08.474337 (+ 13us) server_negotiation.cc:408] Connection header received
0128 15:02:08.474368 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:08.474370 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:08.474407 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:08.474468 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:08.475119 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.475715 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:08.476329 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:08.476549 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:08.477388 (+ 839us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:08.477408 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:08.477411 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:08.477434 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:08.478579 (+ 1145us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:08.478909 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:08.478911 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:08.478913 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:08.478962 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:08.479152 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:08.479155 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:08.479157 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:08.479287 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:08.479402 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:08.479596 (+ 194us) server_negotiation.cc:300] Negotiation successful
0128 15:02:08.479721 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260128 15:02:08.480180 14801 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38776
I20260128 15:02:08.481909 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 14877
I20260128 15:02:08.490559 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 15019
I20260128 15:02:08.495770 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 15161
I20260128 15:02:08.500296 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 14771
2026-01-28T15:02:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3502 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:02:08 dist-test-slave-m28t krb5kdc[15382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:02:08 dist-test-slave-m28t krb5kdc[15382](info): set up 2 sockets
Jan 28 15:02:08 dist-test-slave-m28t krb5kdc[15382](info): commencing operation
krb5kdc: starting...
W20260128 15:02:10.320143 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.799s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Jan 28 15:02:10 dist-test-slave-m28t krb5kdc[15382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:02:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:10.440630 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:33355
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:46783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:33355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:10.540747 15398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:10.540975 15398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:10.541030 15398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:10.544274 15398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:10.544338 15398 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260128 15:02:10.544355 15398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:10.544375 15398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:10.544394 15398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:10.544445 15398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:02:10.548658 15398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:46783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:33355
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:33355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:10.549651 15398 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:10.550469 15398 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:10.555862 15406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:10.555868 15403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:10.555868 15404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:10.556008 15398 server_base.cc:1060] running on GCE node
I20260128 15:02:10.556599 15398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:10.557426 15398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:10.558573 15398 hybrid_clock.cc:648] HybridClock initialized: now 1769612530558560 us; error 29 us; skew 500 ppm
Jan 28 15:02:10 dist-test-slave-m28t krb5kdc[15382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612530, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:10.560966 15398 init.cc:377] Logged in from keytab as kudu/127.13.88.190@KRBTEST.COM (short username kudu)
I20260128 15:02:10.561939 15398 webserver.cc:492] Webserver started at http://127.13.88.190:44507/ using document root <none> and password file <none>
I20260128 15:02:10.562418 15398 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:10.562469 15398 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:10.562625 15398 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:10.564211 15398 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "9bb8020800534568b96875932aed51fb"
format_stamp: "Formatted at 2026-01-28 15:02:10 on dist-test-slave-m28t"
server_key: "1a79e7f6184760b3a19be3b87d6a9f52"
server_key_iv: "73e12f592009b9a98b615da6d0508ade"
server_key_version: "encryptionkey@0"
I20260128 15:02:10.564703 15398 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "9bb8020800534568b96875932aed51fb"
format_stamp: "Formatted at 2026-01-28 15:02:10 on dist-test-slave-m28t"
server_key: "1a79e7f6184760b3a19be3b87d6a9f52"
server_key_iv: "73e12f592009b9a98b615da6d0508ade"
server_key_version: "encryptionkey@0"
I20260128 15:02:10.567934 15398 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260128 15:02:10.569916 15413 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:10.570839 15398 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260128 15:02:10.570935 15398 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "9bb8020800534568b96875932aed51fb"
format_stamp: "Formatted at 2026-01-28 15:02:10 on dist-test-slave-m28t"
server_key: "1a79e7f6184760b3a19be3b87d6a9f52"
server_key_iv: "73e12f592009b9a98b615da6d0508ade"
server_key_version: "encryptionkey@0"
I20260128 15:02:10.571004 15398 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:10.592526 15398 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:10.595084 15398 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:10.595232 15398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:10.602622 15398 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:33355
I20260128 15:02:10.602820 15475 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:33355 every 8 connection(s)
I20260128 15:02:10.603611 15398 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:10.605068 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 15398
I20260128 15:02:10.605159 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:10.605370 13666 external_mini_cluster.cc:1468] Setting key 3053cddc326d4a998bb1c9925740b578
I20260128 15:02:10.606491 15476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:10.612078 15476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb: Bootstrap starting.
Jan 28 15:02:10 dist-test-slave-m28t krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:10.614352 15476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:10.614971 15476 log.cc:826] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:10.616847 15476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb: No bootstrap required, opened a new log
I20260128 15:02:10.617475 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:10.606520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.13.88.190:33355)
0128 15:02:10.606902 (+ 382us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:10.606913 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:10.606951 (+ 38us) server_negotiation.cc:408] Connection header received
0128 15:02:10.607718 (+ 767us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:10.607740 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:10.608086 (+ 346us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:10.608441 (+ 355us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:10.609466 (+ 1025us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:10.610564 (+ 1098us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:10.611200 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:10.611462 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:10.613578 (+ 2116us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:10.613617 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:10.613627 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:10.613659 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:10.615203 (+ 1544us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:10.615593 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:10.615599 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:10.615604 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:10.615697 (+ 93us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:10.615920 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:10.615922 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:10.615924 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:10.616222 (+ 298us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:10.616373 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:10.616645 (+ 272us) server_negotiation.cc:300] Negotiation successful
0128 15:02:10.616885 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":113,"threads_started":1}
I20260128 15:02:10.619905 15476 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb8020800534568b96875932aed51fb" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 33355 } }
I20260128 15:02:10.620074 15476 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:10.620129 15476 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb8020800534568b96875932aed51fb, State: Initialized, Role: FOLLOWER
I20260128 15:02:10.620520 15476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb8020800534568b96875932aed51fb" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 33355 } }
I20260128 15:02:10.620641 15476 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:10.620697 15476 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:10.620783 15476 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:10.621510 15476 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb8020800534568b96875932aed51fb" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 33355 } }
I20260128 15:02:10.621838 15476 leader_election.cc:304] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bb8020800534568b96875932aed51fb; no voters:
I20260128 15:02:10.622128 15476 leader_election.cc:290] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:10.622213 15481 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:10.622408 15481 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [term 1 LEADER]: Becoming Leader. State: Replica: 9bb8020800534568b96875932aed51fb, State: Running, Role: LEADER
I20260128 15:02:10.622697 15481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb8020800534568b96875932aed51fb" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 33355 } }
I20260128 15:02:10.623144 15476 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:10.624127 15482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bb8020800534568b96875932aed51fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb8020800534568b96875932aed51fb" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 33355 } } }
I20260128 15:02:10.624269 15482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:10.624584 15483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bb8020800534568b96875932aed51fb. Latest consensus state: current_term: 1 leader_uuid: "9bb8020800534568b96875932aed51fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb8020800534568b96875932aed51fb" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 33355 } } }
I20260128 15:02:10.624676 15490 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:10.625180 15483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:10.627292 15490 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:10.632091 15490 catalog_manager.cc:1357] Generated new cluster ID: a3fbc377724640a09740ea26e3dcb984
I20260128 15:02:10.632153 15490 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:10.646311 15490 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:10.647140 15490 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:10.653898 15490 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb: Generated new TSK 0
I20260128 15:02:10.654518 15490 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:10.697105 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:33355
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:46783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:10.797917 15504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:10.798126 15504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:10.798200 15504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:10.801476 15504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:10.801539 15504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:10.801618 15504 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260128 15:02:10.801644 15504 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 15:02:10.801693 15504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:02:10.805831 15504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:46783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.88.190:33355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:10.806833 15504 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:10.807621 15504 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:10.813872 15509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:10.813953 15504 server_base.cc:1060] running on GCE node
W20260128 15:02:10.813872 15510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:10.813881 15512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:10.814401 15504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:10.814895 15504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:10.816033 15504 hybrid_clock.cc:648] HybridClock initialized: now 1769612530816023 us; error 24 us; skew 500 ppm
Jan 28 15:02:10 dist-test-slave-m28t krb5kdc[15382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612530, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:10.818411 15504 init.cc:377] Logged in from keytab as kudu/127.13.88.129@KRBTEST.COM (short username kudu)
I20260128 15:02:10.819375 15504 webserver.cc:492] Webserver started at http://127.13.88.129:33115/ using document root <none> and password file <none>
I20260128 15:02:10.819885 15504 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:10.819931 15504 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:10.820097 15504 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:10.821781 15504 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "f365650f1de74e8ca759210dba0d2865"
format_stamp: "Formatted at 2026-01-28 15:02:10 on dist-test-slave-m28t"
server_key: "37cb35b0c388d6cf1720d9dba54cbf11"
server_key_iv: "c44e9359e4c95a209cf1b744067ae920"
server_key_version: "encryptionkey@0"
I20260128 15:02:10.822212 15504 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "f365650f1de74e8ca759210dba0d2865"
format_stamp: "Formatted at 2026-01-28 15:02:10 on dist-test-slave-m28t"
server_key: "37cb35b0c388d6cf1720d9dba54cbf11"
server_key_iv: "c44e9359e4c95a209cf1b744067ae920"
server_key_version: "encryptionkey@0"
I20260128 15:02:10.825407 15504 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260128 15:02:10.827433 15519 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:10.828373 15504 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260128 15:02:10.828493 15504 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "f365650f1de74e8ca759210dba0d2865"
format_stamp: "Formatted at 2026-01-28 15:02:10 on dist-test-slave-m28t"
server_key: "37cb35b0c388d6cf1720d9dba54cbf11"
server_key_iv: "c44e9359e4c95a209cf1b744067ae920"
server_key_version: "encryptionkey@0"
I20260128 15:02:10.828574 15504 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:10.837581 15504 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:10.839805 15504 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:10.839949 15504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:10.841298 15504 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:10.841344 15504 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:10.841392 15504 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:10.841420 15504 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 28 15:02:10 dist-test-slave-m28t krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612530, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:10.851991 15504 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:35019
I20260128 15:02:10.852030 15634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:35019 every 8 connection(s)
I20260128 15:02:10.852443 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:10.841946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:50981 (local address 127.13.88.190:33355)
0128 15:02:10.842055 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:10.842058 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:10.842841 (+ 783us) server_negotiation.cc:408] Connection header received
0128 15:02:10.843697 (+ 856us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:10.843702 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:10.843763 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:10.843852 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:10.845225 (+ 1373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:10.846004 (+ 779us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:10.846719 (+ 715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:10.846906 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:10.849271 (+ 2365us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:10.849290 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:10.849292 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:10.849320 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:10.850642 (+ 1322us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:10.851071 (+ 429us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:10.851074 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:10.851075 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:10.851118 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:10.851352 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:10.851356 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:10.851358 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:10.851513 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:10.851598 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:10.852141 (+ 543us) server_negotiation.cc:300] Negotiation successful
0128 15:02:10.852267 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260128 15:02:10.852957 15504 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:02:10.853072 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:10.842270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:33355 (local address 127.13.88.129:50981)
0128 15:02:10.842707 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:10.842748 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:10.843483 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:10.843986 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:10.843997 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:10.844513 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:10.845091 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:10.845104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:10.846129 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:10.846132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:10.846596 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:10.846604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:10.846774 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:10.847388 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:10.847422 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:10.849134 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:10.850755 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:10.850760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:10.850770 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:10.850982 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:10.851191 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:10.851194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:10.851196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:10.851272 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:10.851609 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:10.851615 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:10.851889 (+ 274us) client_negotiation.cc:769] Sending connection context
0128 15:02:10.852093 (+ 204us) client_negotiation.cc:240] Negotiation successful
0128 15:02:10.852333 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":145,"threads_started":1}
I20260128 15:02:10.854332 15635 heartbeater.cc:344] Connected to a master server at 127.13.88.190:33355
I20260128 15:02:10.854481 15635 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:10.855044 15635 heartbeater.cc:507] Master 127.13.88.190:33355 requested a full tablet report, sending...
I20260128 15:02:10.856568 15430 ts_manager.cc:194] Registered new tserver with Master: f365650f1de74e8ca759210dba0d2865 (127.13.88.129:35019)
I20260128 15:02:10.857682 15430 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.129@KRBTEST.COM'} at 127.13.88.129:50981
I20260128 15:02:10.861668 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 15504
I20260128 15:02:10.861778 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:02:10.861840 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:10.856453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:48467 (local address 127.13.88.190:33355)
0128 15:02:10.856563 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:10.856567 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:10.856782 (+ 215us) server_negotiation.cc:408] Connection header received
0128 15:02:10.856878 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:10.856882 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:10.856923 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:10.856981 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:10.857457 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:10.857999 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:10.858630 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:10.858805 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:10.859507 (+ 702us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:10.859523 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:10.859526 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:10.859546 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:10.860625 (+ 1079us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:10.860963 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:10.860967 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:10.860968 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:10.861007 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:10.861195 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:10.861198 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:10.861200 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:10.861322 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:10.861402 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:10.861618 (+ 216us) server_negotiation.cc:300] Negotiation successful
0128 15:02:10.861714 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260128 15:02:10.861858 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:10.856361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:33355 (local address 127.13.88.129:48467)
0128 15:02:10.856698 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:10.856711 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:10.856809 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:10.857005 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:10.857008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:10.857163 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:10.857354 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:10.857359 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:10.858106 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:10.858109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:10.858529 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:10.858538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:10.858635 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:10.859129 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:10.859141 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:10.859389 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:10.860725 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:10.860729 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:10.860731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:10.860881 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:10.861078 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:10.861081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:10.861083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:10.861124 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:10.861402 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:10.861406 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:10.861478 (+ 72us) client_negotiation.cc:769] Sending connection context
0128 15:02:10.861562 (+ 84us) client_negotiation.cc:240] Negotiation successful
0128 15:02:10.861677 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":104,"threads_started":1}
I20260128 15:02:10.862028 13666 external_mini_cluster.cc:1468] Setting key 1de11f9ae9a2fce53d0af3f18f66953b
WARNING: no policy specified for kudu/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:10.900900 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:33355
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:46783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:11.000169 15646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:11.000375 15646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:11.000463 15646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:11.003611 15646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:11.003671 15646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:11.003743 15646 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260128 15:02:11.003773 15646 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 15:02:11.003825 15646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:02:11.007851 15646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:46783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.88.190:33355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:11.008867 15646 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:11.009651 15646 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:11.015842 15651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:11.015879 15652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:11.015879 15654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:11.016026 15646 server_base.cc:1060] running on GCE node
I20260128 15:02:11.016505 15646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:11.016983 15646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:11.018152 15646 hybrid_clock.cc:648] HybridClock initialized: now 1769612531018140 us; error 28 us; skew 500 ppm
Jan 28 15:02:11 dist-test-slave-m28t krb5kdc[15382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612531, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:11.020634 15646 init.cc:377] Logged in from keytab as kudu/127.13.88.130@KRBTEST.COM (short username kudu)
I20260128 15:02:11.021579 15646 webserver.cc:492] Webserver started at http://127.13.88.130:45109/ using document root <none> and password file <none>
I20260128 15:02:11.022051 15646 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:11.022092 15646 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:11.022243 15646 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:11.023825 15646 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "a72f9aaff93c4696bd1bdd1b78e0f280"
format_stamp: "Formatted at 2026-01-28 15:02:11 on dist-test-slave-m28t"
server_key: "8222460d6605339dc76fc007ed77f31b"
server_key_iv: "cc331276a6f0e6e92a7cee90d0623889"
server_key_version: "encryptionkey@0"
I20260128 15:02:11.024214 15646 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "a72f9aaff93c4696bd1bdd1b78e0f280"
format_stamp: "Formatted at 2026-01-28 15:02:11 on dist-test-slave-m28t"
server_key: "8222460d6605339dc76fc007ed77f31b"
server_key_iv: "cc331276a6f0e6e92a7cee90d0623889"
server_key_version: "encryptionkey@0"
I20260128 15:02:11.027354 15646 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260128 15:02:11.029362 15661 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:11.030293 15646 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:02:11.030383 15646 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "a72f9aaff93c4696bd1bdd1b78e0f280"
format_stamp: "Formatted at 2026-01-28 15:02:11 on dist-test-slave-m28t"
server_key: "8222460d6605339dc76fc007ed77f31b"
server_key_iv: "cc331276a6f0e6e92a7cee90d0623889"
server_key_version: "encryptionkey@0"
I20260128 15:02:11.030454 15646 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:11.052521 15646 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:11.054937 15646 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:11.055114 15646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:11.056630 15646 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:11.056681 15646 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:11.056721 15646 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:11.056751 15646 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 28 15:02:11 dist-test-slave-m28t krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612531, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:11.067025 15646 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:40711
I20260128 15:02:11.067040 15776 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:40711 every 8 connection(s)
I20260128 15:02:11.068126 15646 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
I20260128 15:02:11.068382 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.057200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:37951 (local address 127.13.88.190:33355)
0128 15:02:11.057337 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.057342 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.058110 (+ 768us) server_negotiation.cc:408] Connection header received
0128 15:02:11.059154 (+ 1044us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.059157 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.059208 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.059278 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:11.061015 (+ 1737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.061798 (+ 783us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.062515 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.062692 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.064962 (+ 2270us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:11.064985 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:11.064988 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:11.065019 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:11.066498 (+ 1479us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:11.067063 (+ 565us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:11.067067 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:11.067069 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:11.067126 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:11.067404 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:11.067407 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:11.067408 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:11.067558 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:11.067628 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.068163 (+ 535us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.068265 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260128 15:02:11.069176 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.057468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:33355 (local address 127.13.88.130:37951)
0128 15:02:11.057957 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:11.058004 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:11.058937 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:11.059465 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:11.059478 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:11.060032 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:11.060845 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.060864 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.061918 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.061922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:11.062393 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.062401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.062602 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.063211 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:11.063245 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:11.064808 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:11.066613 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:11.066618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:11.066628 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:11.066946 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:11.067220 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:11.067224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:11.067226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:11.067326 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:11.067646 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:11.067652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:11.067934 (+ 282us) client_negotiation.cc:769] Sending connection context
0128 15:02:11.068150 (+ 216us) client_negotiation.cc:240] Negotiation successful
0128 15:02:11.068410 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":108,"threads_started":1}
I20260128 15:02:11.071839 15777 heartbeater.cc:344] Connected to a master server at 127.13.88.190:33355
I20260128 15:02:11.071974 15777 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:11.072436 15777 heartbeater.cc:507] Master 127.13.88.190:33355 requested a full tablet report, sending...
I20260128 15:02:11.073361 15430 ts_manager.cc:194] Registered new tserver with Master: a72f9aaff93c4696bd1bdd1b78e0f280 (127.13.88.130:40711)
I20260128 15:02:11.073922 15430 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.130@KRBTEST.COM'} at 127.13.88.130:37951
I20260128 15:02:11.075515 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 15646
I20260128 15:02:11.075629 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:02:11.075876 13666 external_mini_cluster.cc:1468] Setting key a8086c274c2f19b7ed45ea2dc75dd931
I20260128 15:02:11.078322 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.072063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:50485 (local address 127.13.88.190:33355)
0128 15:02:11.072177 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.072181 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.072644 (+ 463us) server_negotiation.cc:408] Connection header received
0128 15:02:11.072724 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.072727 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.072774 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.072853 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:11.073380 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.073877 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.074591 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.074755 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.075481 (+ 726us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:11.075496 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:11.075498 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:11.075519 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:11.076980 (+ 1461us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:11.077382 (+ 402us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:11.077386 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:11.077388 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:11.077430 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:11.077628 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:11.077631 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:11.077634 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:11.077785 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:11.077868 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.078098 (+ 230us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.078194 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260128 15:02:11.078336 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.071970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:33355 (local address 127.13.88.130:50485)
0128 15:02:11.072571 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:11.072583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:11.072663 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:11.072860 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:11.072862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:11.073039 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:11.073268 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.073275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.073986 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.073990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:11.074490 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.074499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.074585 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.075039 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:11.075052 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:11.075359 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:11.077070 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:11.077073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:11.077076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:11.077298 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:11.077503 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:11.077506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:11.077509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:11.077558 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:11.077870 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:11.077873 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:11.077951 (+ 78us) client_negotiation.cc:769] Sending connection context
0128 15:02:11.078038 (+ 87us) client_negotiation.cc:240] Negotiation successful
0128 15:02:11.078161 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:11.115183 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:33355
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:46783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:11.216490 15788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:11.216686 15788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:11.216768 15788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:11.219904 15788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:11.219960 15788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:11.220017 15788 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260128 15:02:11.220046 15788 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 15:02:11.220106 15788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:02:11.224197 15788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:46783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.88.190:33355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:11.225225 15788 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:11.226009 15788 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:11.231979 15796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:11.232048 15793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:11.232019 15794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:11.232146 15788 server_base.cc:1060] running on GCE node
I20260128 15:02:11.232576 15788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:11.233047 15788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:11.234189 15788 hybrid_clock.cc:648] HybridClock initialized: now 1769612531234175 us; error 30 us; skew 500 ppm
Jan 28 15:02:11 dist-test-slave-m28t krb5kdc[15382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612531, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:11.236639 15788 init.cc:377] Logged in from keytab as kudu/127.13.88.131@KRBTEST.COM (short username kudu)
I20260128 15:02:11.237583 15788 webserver.cc:492] Webserver started at http://127.13.88.131:38215/ using document root <none> and password file <none>
I20260128 15:02:11.238080 15788 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:11.238127 15788 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:11.238286 15788 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:11.239878 15788 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "fd1d86066bc449b2b6527f6056048470"
format_stamp: "Formatted at 2026-01-28 15:02:11 on dist-test-slave-m28t"
server_key: "5373c9ec2543e9e10b64113bf5eb8ed0"
server_key_iv: "048beeb4414069cfbcb743a70efb4822"
server_key_version: "encryptionkey@0"
I20260128 15:02:11.240316 15788 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "fd1d86066bc449b2b6527f6056048470"
format_stamp: "Formatted at 2026-01-28 15:02:11 on dist-test-slave-m28t"
server_key: "5373c9ec2543e9e10b64113bf5eb8ed0"
server_key_iv: "048beeb4414069cfbcb743a70efb4822"
server_key_version: "encryptionkey@0"
I20260128 15:02:11.243396 15788 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260128 15:02:11.245388 15803 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:11.246338 15788 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:02:11.246431 15788 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "fd1d86066bc449b2b6527f6056048470"
format_stamp: "Formatted at 2026-01-28 15:02:11 on dist-test-slave-m28t"
server_key: "5373c9ec2543e9e10b64113bf5eb8ed0"
server_key_iv: "048beeb4414069cfbcb743a70efb4822"
server_key_version: "encryptionkey@0"
I20260128 15:02:11.246508 15788 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:11.266083 15788 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:11.268596 15788 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:11.268762 15788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:11.270121 15788 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:11.270169 15788 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:11.270218 15788 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:11.270278 15788 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 28 15:02:11 dist-test-slave-m28t krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612531, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:11.280267 15788 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:42773
I20260128 15:02:11.280336 15918 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:42773 every 8 connection(s)
I20260128 15:02:11.281185 15788 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
I20260128 15:02:11.281347 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.270706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:33401 (local address 127.13.88.190:33355)
0128 15:02:11.270828 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.270832 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.271612 (+ 780us) server_negotiation.cc:408] Connection header received
0128 15:02:11.272457 (+ 845us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.272461 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.272530 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.272606 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:11.274020 (+ 1414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.274641 (+ 621us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.275504 (+ 863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.275699 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.277984 (+ 2285us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:11.278006 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:11.278009 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:11.278040 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:11.279586 (+ 1546us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:11.280002 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:11.280005 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:11.280007 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:11.280053 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:11.280345 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:11.280349 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:11.280351 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:11.280543 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:11.280635 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.281077 (+ 442us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.281194 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260128 15:02:11.281946 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.271030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:33355 (local address 127.13.88.131:33401)
0128 15:02:11.271478 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:11.271512 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:11.272235 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:11.272779 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:11.272790 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:11.273274 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:11.273873 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.273886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.274756 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.274761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:11.275358 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.275369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.275614 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.276174 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:11.276208 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:11.277840 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:11.279689 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:11.279693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:11.279701 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:11.279907 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:11.280155 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:11.280158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:11.280160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:11.280238 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:11.280643 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:11.280649 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:11.280855 (+ 206us) client_negotiation.cc:769] Sending connection context
0128 15:02:11.281023 (+ 168us) client_negotiation.cc:240] Negotiation successful
0128 15:02:11.281253 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":147,"threads_started":1}
I20260128 15:02:11.283161 15919 heartbeater.cc:344] Connected to a master server at 127.13.88.190:33355
I20260128 15:02:11.283299 15919 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:11.283735 15919 heartbeater.cc:507] Master 127.13.88.190:33355 requested a full tablet report, sending...
I20260128 15:02:11.284636 15429 ts_manager.cc:194] Registered new tserver with Master: fd1d86066bc449b2b6527f6056048470 (127.13.88.131:42773)
I20260128 15:02:11.285264 15429 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.131@KRBTEST.COM'} at 127.13.88.131:33401
I20260128 15:02:11.289664 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 15788
I20260128 15:02:11.289757 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:02:11.289970 13666 external_mini_cluster.cc:1468] Setting key 7959e3c60f69c3cb214e3b11dfc1a4fa
I20260128 15:02:11.290925 15925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.284906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:33355 (local address 127.13.88.131:38229)
0128 15:02:11.285139 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:11.285151 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:11.285236 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:11.285440 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:11.285443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:11.285618 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:11.285860 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.285867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.286505 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.286509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:11.286886 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.286893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.286986 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.287464 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:11.287484 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:11.287732 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:11.289045 (+ 1313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:11.289049 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:11.289051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:11.289206 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:11.289402 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:11.289405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:11.289407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:11.289447 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:11.290507 (+ 1060us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:11.290510 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:11.290583 (+ 73us) client_negotiation.cc:769] Sending connection context
0128 15:02:11.290686 (+ 103us) client_negotiation.cc:240] Negotiation successful
0128 15:02:11.290792 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
I20260128 15:02:11.290973 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.284971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:38229 (local address 127.13.88.190:33355)
0128 15:02:11.285074 (+ 103us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.285078 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.285208 (+ 130us) server_negotiation.cc:408] Connection header received
0128 15:02:11.285304 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.285307 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.285351 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.285415 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:11.285962 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.286405 (+ 443us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.286982 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.287154 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.287838 (+ 684us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:11.287854 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:11.287857 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:11.287879 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:11.288948 (+ 1069us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:11.289286 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:11.289289 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:11.289290 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:11.289329 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:11.290238 (+ 909us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:11.290241 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:11.290243 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:11.290411 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:11.290498 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.290749 (+ 251us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.290857 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260128 15:02:11.291803 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 28 15:02:11 dist-test-slave-m28t krb5kdc[15382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 28 15:02:11 dist-test-slave-m28t krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:11.312645 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.306301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58334 (local address 127.13.88.190:33355)
0128 15:02:11.306423 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.306426 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.306438 (+ 12us) server_negotiation.cc:408] Connection header received
0128 15:02:11.306483 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.306486 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.306528 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.306593 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:11.307307 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.307794 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.308442 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.308577 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.310473 (+ 1896us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:11.310491 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:11.310493 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:11.310518 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:11.311558 (+ 1040us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:11.311865 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:11.311867 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:11.311868 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:11.311902 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:11.312038 (+ 136us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:11.312040 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:11.312041 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:11.312163 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:11.312249 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.312376 (+ 127us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.312497 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260128 15:02:11.314879 15430 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 15:02:11.317207 15430 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:02:11.331650 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.325273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:40711 (local address 127.0.0.1:41870)
0128 15:02:11.326957 (+ 1684us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:11.326996 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:11.327140 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:11.327883 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:11.327890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:11.327914 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:11.328209 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.328218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.330304 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.330308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:11.331157 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.331169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.331253 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.331308 (+ 55us) client_negotiation.cc:769] Sending connection context
0128 15:02:11.331383 (+ 75us) client_negotiation.cc:240] Negotiation successful
0128 15:02:11.331493 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1524,"thread_start_us":86,"threads_started":1}
I20260128 15:02:11.332856 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.326485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:42773 (local address 127.0.0.1:44792)
0128 15:02:11.327877 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:11.327890 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:11.327964 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:11.328562 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:11.328565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:11.328584 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:11.328761 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.328766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.331594 (+ 2828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.331598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:11.332470 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.332481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.332572 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.332593 (+ 21us) client_negotiation.cc:769] Sending connection context
0128 15:02:11.332642 (+ 49us) client_negotiation.cc:240] Negotiation successful
0128 15:02:11.332707 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1314,"thread_start_us":102,"threads_started":1}
I20260128 15:02:11.333048 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.326058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.13.88.130:40711)
0128 15:02:11.326392 (+ 334us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.326399 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.327266 (+ 867us) server_negotiation.cc:408] Connection header received
0128 15:02:11.327354 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.327360 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.327558 (+ 198us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.327693 (+ 135us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:11.328335 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.329574 (+ 1239us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.332030 (+ 2456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.332644 (+ 614us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.332767 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.332832 (+ 65us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.332918 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":109,"threads_started":1}
I20260128 15:02:11.333714 15941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.327826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42412 (local address 127.13.88.129:35019)
0128 15:02:11.328497 (+ 671us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.328502 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.329044 (+ 542us) server_negotiation.cc:408] Connection header received
0128 15:02:11.329262 (+ 218us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.329267 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.329431 (+ 164us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.329557 (+ 126us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:11.330158 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.331281 (+ 1123us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.332637 (+ 1356us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.333162 (+ 525us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.333273 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.333502 (+ 229us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.333595 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":83,"threads_started":1}
I20260128 15:02:11.334105 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.327658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:35019 (local address 127.0.0.1:42412)
0128 15:02:11.328963 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:11.328976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:11.329110 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:11.329755 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:11.329759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:11.329778 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:11.330006 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.330013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.331414 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.331417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:11.332470 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.332485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.333334 (+ 849us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.333349 (+ 15us) client_negotiation.cc:769] Sending connection context
0128 15:02:11.333901 (+ 552us) client_negotiation.cc:240] Negotiation successful
0128 15:02:11.333957 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1240,"thread_start_us":86,"threads_started":1}
I20260128 15:02:11.334637 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.326890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44792 (local address 127.13.88.131:42773)
0128 15:02:11.327333 (+ 443us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.327339 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.328051 (+ 712us) server_negotiation.cc:408] Connection header received
0128 15:02:11.328133 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.328138 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.328303 (+ 165us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.328451 (+ 148us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:11.328875 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.331480 (+ 2605us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.333161 (+ 1681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.334268 (+ 1107us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.334364 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.334422 (+ 58us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.334511 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":118,"threads_started":1}
I20260128 15:02:11.336203 15711 tablet_service.cc:1505] Processing CreateTablet for tablet 2c55e926c50c4c96b0c451a78c052464 (DEFAULT_TABLE table=test-table [id=096721b8bb3345f09cd6a631bc72361d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:11.336516 15569 tablet_service.cc:1505] Processing CreateTablet for tablet 2c55e926c50c4c96b0c451a78c052464 (DEFAULT_TABLE table=test-table [id=096721b8bb3345f09cd6a631bc72361d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:11.337267 15569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c55e926c50c4c96b0c451a78c052464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:11.337275 15711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c55e926c50c4c96b0c451a78c052464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:11.342758 15853 tablet_service.cc:1505] Processing CreateTablet for tablet 2c55e926c50c4c96b0c451a78c052464 (DEFAULT_TABLE table=test-table [id=096721b8bb3345f09cd6a631bc72361d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:11.343691 15853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c55e926c50c4c96b0c451a78c052464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:11.344015 15942 tablet_bootstrap.cc:492] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865: Bootstrap starting.
I20260128 15:02:11.346263 15942 tablet_bootstrap.cc:654] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:11.347530 15942 log.cc:826] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:11.348639 15943 tablet_bootstrap.cc:492] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280: Bootstrap starting.
I20260128 15:02:11.349521 15942 tablet_bootstrap.cc:492] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865: No bootstrap required, opened a new log
I20260128 15:02:11.349728 15942 ts_tablet_manager.cc:1403] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260128 15:02:11.350862 15943 tablet_bootstrap.cc:654] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:11.350880 15945 tablet_bootstrap.cc:492] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470: Bootstrap starting.
I20260128 15:02:11.352051 15943 log.cc:826] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:11.352895 15945 tablet_bootstrap.cc:654] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:11.352957 15942 raft_consensus.cc:359] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.353169 15942 raft_consensus.cc:385] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:11.353220 15942 raft_consensus.cc:740] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f365650f1de74e8ca759210dba0d2865, State: Initialized, Role: FOLLOWER
I20260128 15:02:11.353921 15943 tablet_bootstrap.cc:492] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280: No bootstrap required, opened a new log
I20260128 15:02:11.354007 15945 log.cc:826] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:11.354104 15943 ts_tablet_manager.cc:1403] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260128 15:02:11.354816 15942 consensus_queue.cc:260] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.355731 15942 ts_tablet_manager.cc:1434] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20260128 15:02:11.355755 15945 tablet_bootstrap.cc:492] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470: No bootstrap required, opened a new log
I20260128 15:02:11.355922 15945 ts_tablet_manager.cc:1403] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260128 15:02:11.356097 15635 heartbeater.cc:499] Master 127.13.88.190:33355 was elected leader, sending a full tablet report...
I20260128 15:02:11.357555 15943 raft_consensus.cc:359] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.357833 15943 raft_consensus.cc:385] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:11.357932 15943 raft_consensus.cc:740] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72f9aaff93c4696bd1bdd1b78e0f280, State: Initialized, Role: FOLLOWER
I20260128 15:02:11.358420 15943 consensus_queue.cc:260] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.358764 15945 raft_consensus.cc:359] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.359017 15945 raft_consensus.cc:385] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:11.359102 15945 raft_consensus.cc:740] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1d86066bc449b2b6527f6056048470, State: Initialized, Role: FOLLOWER
I20260128 15:02:11.359280 15943 ts_tablet_manager.cc:1434] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260128 15:02:11.359551 15945 consensus_queue.cc:260] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.359658 15777 heartbeater.cc:499] Master 127.13.88.190:33355 was elected leader, sending a full tablet report...
I20260128 15:02:11.360404 15945 ts_tablet_manager.cc:1434] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260128 15:02:11.360872 15919 heartbeater.cc:499] Master 127.13.88.190:33355 was elected leader, sending a full tablet report...
W20260128 15:02:11.532084 15920 tablet.cc:2378] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 15:02:11.539809 15948 raft_consensus.cc:493] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:02:11.539994 15948 raft_consensus.cc:515] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.541214 15948 leader_election.cc:290] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a72f9aaff93c4696bd1bdd1b78e0f280 (127.13.88.130:40711), fd1d86066bc449b2b6527f6056048470 (127.13.88.131:42773)
I20260128 15:02:11.544443 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.541418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:40711 (local address 127.13.88.129:33119)
0128 15:02:11.541806 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:11.541821 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:11.541966 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:11.542256 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:11.542260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:11.542276 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:11.542548 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.542556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.543282 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.543285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:11.544020 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.544029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.544140 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.544162 (+ 22us) client_negotiation.cc:769] Sending connection context
0128 15:02:11.544214 (+ 52us) client_negotiation.cc:240] Negotiation successful
0128 15:02:11.544269 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":72,"threads_started":1}
I20260128 15:02:11.544970 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.541536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:33119 (local address 127.13.88.130:40711)
0128 15:02:11.541669 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.541674 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.541888 (+ 214us) server_negotiation.cc:408] Connection header received
0128 15:02:11.542107 (+ 219us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.542110 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.542156 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.542228 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:11.542674 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.543182 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.544155 (+ 973us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.544697 (+ 542us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.544747 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.544798 (+ 51us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.544852 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260128 15:02:11.545431 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.541760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:49971 (local address 127.13.88.131:42773)
0128 15:02:11.541872 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.541877 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.542013 (+ 136us) server_negotiation.cc:408] Connection header received
0128 15:02:11.542195 (+ 182us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.542199 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.542251 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.542313 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:11.542758 (+ 445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.543529 (+ 771us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.544639 (+ 1110us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.545011 (+ 372us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.545047 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.545253 (+ 206us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.545327 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260128 15:02:11.545651 15731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c55e926c50c4c96b0c451a78c052464" candidate_uuid: "f365650f1de74e8ca759210dba0d2865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" is_pre_election: true
I20260128 15:02:11.545764 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.541494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:42773 (local address 127.13.88.129:49971)
0128 15:02:11.541920 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:11.541935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:11.542044 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:11.542348 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:11.542351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:11.542369 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:11.542578 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.542583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.543656 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.543660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:11.544500 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.544509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.545112 (+ 603us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.545128 (+ 16us) client_negotiation.cc:769] Sending connection context
0128 15:02:11.545600 (+ 472us) client_negotiation.cc:240] Negotiation successful
0128 15:02:11.545665 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":83,"threads_started":1}
I20260128 15:02:11.545948 15731 raft_consensus.cc:2468] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f365650f1de74e8ca759210dba0d2865 in term 0.
I20260128 15:02:11.546325 15873 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c55e926c50c4c96b0c451a78c052464" candidate_uuid: "f365650f1de74e8ca759210dba0d2865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd1d86066bc449b2b6527f6056048470" is_pre_election: true
I20260128 15:02:11.546444 15523 leader_election.cc:304] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72f9aaff93c4696bd1bdd1b78e0f280, f365650f1de74e8ca759210dba0d2865; no voters:
I20260128 15:02:11.546576 15873 raft_consensus.cc:2468] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f365650f1de74e8ca759210dba0d2865 in term 0.
I20260128 15:02:11.546741 15948 raft_consensus.cc:2804] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:02:11.546811 15948 raft_consensus.cc:493] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:02:11.546847 15948 raft_consensus.cc:3060] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:11.547995 15948 raft_consensus.cc:515] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.548381 15948 leader_election.cc:290] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [CANDIDATE]: Term 1 election: Requested vote from peers a72f9aaff93c4696bd1bdd1b78e0f280 (127.13.88.130:40711), fd1d86066bc449b2b6527f6056048470 (127.13.88.131:42773)
I20260128 15:02:11.548746 15873 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c55e926c50c4c96b0c451a78c052464" candidate_uuid: "f365650f1de74e8ca759210dba0d2865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd1d86066bc449b2b6527f6056048470"
I20260128 15:02:11.548763 15731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c55e926c50c4c96b0c451a78c052464" candidate_uuid: "f365650f1de74e8ca759210dba0d2865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280"
I20260128 15:02:11.548868 15873 raft_consensus.cc:3060] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:11.548877 15731 raft_consensus.cc:3060] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:11.549818 15731 raft_consensus.cc:2468] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f365650f1de74e8ca759210dba0d2865 in term 1.
I20260128 15:02:11.550062 15873 raft_consensus.cc:2468] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f365650f1de74e8ca759210dba0d2865 in term 1.
I20260128 15:02:11.550119 15523 leader_election.cc:304] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72f9aaff93c4696bd1bdd1b78e0f280, f365650f1de74e8ca759210dba0d2865; no voters:
I20260128 15:02:11.550357 15948 raft_consensus.cc:2804] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:11.550545 15948 raft_consensus.cc:697] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [term 1 LEADER]: Becoming Leader. State: Replica: f365650f1de74e8ca759210dba0d2865, State: Running, Role: LEADER
I20260128 15:02:11.550913 15948 consensus_queue.cc:237] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.554499 15430 catalog_manager.cc:5654] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 reported cstate change: term changed from 0 to 1, leader changed from <none> to f365650f1de74e8ca759210dba0d2865 (127.13.88.129). New cstate: current_term: 1 leader_uuid: "f365650f1de74e8ca759210dba0d2865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } health_report { overall_health: UNKNOWN } } }
W20260128 15:02:11.569314 15778 tablet.cc:2378] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jan 28 15:02:11 dist-test-slave-m28t krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612530, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:11.592851 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.585964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:33355 (local address 127.0.0.1:58348)
0128 15:02:11.586212 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:11.586224 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:11.586328 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:11.586550 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:11.586553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:11.586976 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:11.587175 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.587183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.587863 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.587867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:11.588227 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.588233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.588329 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.588910 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:11.588923 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:11.590226 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:11.591527 (+ 1301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:11.591535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:11.591541 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:11.591750 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:11.591945 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:11.591948 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:11.591950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:11.591990 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:11.592279 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:11.592286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:11.592376 (+ 90us) client_negotiation.cc:769] Sending connection context
0128 15:02:11.592502 (+ 126us) client_negotiation.cc:240] Negotiation successful
0128 15:02:11.592643 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
I20260128 15:02:11.592851 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.586024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.13.88.190:33355)
0128 15:02:11.586145 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.586151 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.586289 (+ 138us) server_negotiation.cc:408] Connection header received
0128 15:02:11.586406 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.586409 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.586447 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.586515 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:11.587291 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.587769 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.588322 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.588530 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.590335 (+ 1805us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:11.590348 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:11.590350 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:11.590372 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:11.591427 (+ 1055us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:11.591833 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:11.591836 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:11.591837 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:11.591875 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:11.592057 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:11.592060 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:11.592062 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:11.592189 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:11.592267 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.592553 (+ 286us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.592695 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":39}
I20260128 15:02:11.596288 15430 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:58348:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260128 15:02:11.596957 15430 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:02:11.601547 15569 tablet_service.cc:1505] Processing CreateTablet for tablet 9a0221fc44b94dabbb1e2237ef7a7b82 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4adeded45139471e97a92dcc3e6a5a12]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 15:02:11.601670 15711 tablet_service.cc:1505] Processing CreateTablet for tablet 9a0221fc44b94dabbb1e2237ef7a7b82 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4adeded45139471e97a92dcc3e6a5a12]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 15:02:11.601872 15853 tablet_service.cc:1505] Processing CreateTablet for tablet 9a0221fc44b94dabbb1e2237ef7a7b82 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4adeded45139471e97a92dcc3e6a5a12]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 15:02:11.602211 15569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a0221fc44b94dabbb1e2237ef7a7b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:11.602234 15711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a0221fc44b94dabbb1e2237ef7a7b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:11.602303 15853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a0221fc44b94dabbb1e2237ef7a7b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:11.604789 15943 tablet_bootstrap.cc:492] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280: Bootstrap starting.
W20260128 15:02:11.604804 15636 tablet.cc:2378] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 15:02:11.605103 15942 tablet_bootstrap.cc:492] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865: Bootstrap starting.
I20260128 15:02:11.605437 15945 tablet_bootstrap.cc:492] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470: Bootstrap starting.
I20260128 15:02:11.605755 15943 tablet_bootstrap.cc:654] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:11.606053 15942 tablet_bootstrap.cc:654] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:11.606372 15945 tablet_bootstrap.cc:654] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:11.607275 15942 tablet_bootstrap.cc:492] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865: No bootstrap required, opened a new log
I20260128 15:02:11.607365 15942 ts_tablet_manager.cc:1403] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260128 15:02:11.607657 15945 tablet_bootstrap.cc:492] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470: No bootstrap required, opened a new log
I20260128 15:02:11.607740 15945 ts_tablet_manager.cc:1403] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260128 15:02:11.607836 15942 raft_consensus.cc:359] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.607941 15942 raft_consensus.cc:385] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:11.607980 15942 raft_consensus.cc:740] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f365650f1de74e8ca759210dba0d2865, State: Initialized, Role: FOLLOWER
I20260128 15:02:11.608115 15942 consensus_queue.cc:260] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.608366 15942 ts_tablet_manager.cc:1434] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:11.608376 15945 raft_consensus.cc:359] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.608505 15945 raft_consensus.cc:385] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:11.608541 15945 raft_consensus.cc:740] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1d86066bc449b2b6527f6056048470, State: Initialized, Role: FOLLOWER
I20260128 15:02:11.608603 15948 tablet_replica.cc:442] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } } }
I20260128 15:02:11.608645 15945 consensus_queue.cc:260] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.608695 15948 tablet_replica.cc:445] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:02:11.608997 15950 tablet_replica.cc:442] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } } }
I20260128 15:02:11.609114 15950 tablet_replica.cc:445] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:02:11.609819 15945 ts_tablet_manager.cc:1434] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260128 15:02:11.610499 15943 tablet_bootstrap.cc:492] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280: No bootstrap required, opened a new log
I20260128 15:02:11.610580 15943 ts_tablet_manager.cc:1403] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260128 15:02:11.611017 15943 raft_consensus.cc:359] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.611112 15943 raft_consensus.cc:385] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:11.611142 15943 raft_consensus.cc:740] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72f9aaff93c4696bd1bdd1b78e0f280, State: Initialized, Role: FOLLOWER
I20260128 15:02:11.611255 15943 consensus_queue.cc:260] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.611490 15943 ts_tablet_manager.cc:1434] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:11.611682 15949 tablet_replica.cc:442] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } } }
I20260128 15:02:11.611791 15949 tablet_replica.cc:445] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:02:11.855583 15949 raft_consensus.cc:493] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:02:11.855717 15949 raft_consensus.cc:515] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.856885 15949 leader_election.cc:290] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f365650f1de74e8ca759210dba0d2865 (127.13.88.129:35019), fd1d86066bc449b2b6527f6056048470 (127.13.88.131:42773)
I20260128 15:02:11.860008 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.857040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:35019 (local address 127.13.88.130:55495)
0128 15:02:11.857361 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:11.857375 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:11.857492 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:11.857766 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:11.857769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:11.857787 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:11.858071 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.858078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.858814 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.858817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:11.859634 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.859643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.859746 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.859765 (+ 19us) client_negotiation.cc:769] Sending connection context
0128 15:02:11.859810 (+ 45us) client_negotiation.cc:240] Negotiation successful
0128 15:02:11.859868 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":91,"threads_started":1}
I20260128 15:02:11.860623 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.857204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:55495 (local address 127.13.88.129:35019)
0128 15:02:11.857390 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.857395 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.857435 (+ 40us) server_negotiation.cc:408] Connection header received
0128 15:02:11.857614 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.857617 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.857666 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.857746 (+ 80us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:11.858188 (+ 442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.858698 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.859855 (+ 1157us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.860334 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.860367 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.860411 (+ 44us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.860494 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":52,"threads_started":1}
I20260128 15:02:11.861105 15589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a0221fc44b94dabbb1e2237ef7a7b82" candidate_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f365650f1de74e8ca759210dba0d2865" is_pre_election: true
I20260128 15:02:11.861181 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.857462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:52593 (local address 127.13.88.131:42773)
0128 15:02:11.857605 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.857610 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.857623 (+ 13us) server_negotiation.cc:408] Connection header received
0128 15:02:11.857666 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.857670 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.857715 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.857787 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:11.858226 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.858917 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.860110 (+ 1193us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.860557 (+ 447us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.860587 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.861013 (+ 426us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.861081 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260128 15:02:11.861275 15589 raft_consensus.cc:2468] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72f9aaff93c4696bd1bdd1b78e0f280 in term 0.
I20260128 15:02:11.861328 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.857118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:42773 (local address 127.13.88.130:52593)
0128 15:02:11.857497 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:11.857511 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:11.857598 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:11.857874 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:11.857877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:11.857894 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:11.858088 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.858093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.859021 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.859025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:11.859975 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.859984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.860638 (+ 654us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.860650 (+ 12us) client_negotiation.cc:769] Sending connection context
0128 15:02:11.860971 (+ 321us) client_negotiation.cc:240] Negotiation successful
0128 15:02:11.861032 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":80,"threads_started":1}
I20260128 15:02:11.861759 15664 leader_election.cc:304] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72f9aaff93c4696bd1bdd1b78e0f280, f365650f1de74e8ca759210dba0d2865; no voters:
I20260128 15:02:11.861766 15873 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a0221fc44b94dabbb1e2237ef7a7b82" candidate_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd1d86066bc449b2b6527f6056048470" is_pre_election: true
I20260128 15:02:11.861883 15873 raft_consensus.cc:2468] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72f9aaff93c4696bd1bdd1b78e0f280 in term 0.
I20260128 15:02:11.862004 15949 raft_consensus.cc:2804] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:02:11.862066 15949 raft_consensus.cc:493] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:02:11.862103 15949 raft_consensus.cc:3060] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:11.862913 15949 raft_consensus.cc:515] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.863197 15949 leader_election.cc:290] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [CANDIDATE]: Term 1 election: Requested vote from peers f365650f1de74e8ca759210dba0d2865 (127.13.88.129:35019), fd1d86066bc449b2b6527f6056048470 (127.13.88.131:42773)
I20260128 15:02:11.863508 15873 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a0221fc44b94dabbb1e2237ef7a7b82" candidate_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd1d86066bc449b2b6527f6056048470"
I20260128 15:02:11.863538 15589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9a0221fc44b94dabbb1e2237ef7a7b82" candidate_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f365650f1de74e8ca759210dba0d2865"
I20260128 15:02:11.863612 15873 raft_consensus.cc:3060] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:11.863628 15589 raft_consensus.cc:3060] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:11.864341 15873 raft_consensus.cc:2468] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72f9aaff93c4696bd1bdd1b78e0f280 in term 1.
I20260128 15:02:11.864398 15589 raft_consensus.cc:2468] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72f9aaff93c4696bd1bdd1b78e0f280 in term 1.
I20260128 15:02:11.864643 15663 leader_election.cc:304] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72f9aaff93c4696bd1bdd1b78e0f280, fd1d86066bc449b2b6527f6056048470; no voters:
I20260128 15:02:11.864820 15949 raft_consensus.cc:2804] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:11.864993 15949 raft_consensus.cc:697] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 LEADER]: Becoming Leader. State: Replica: a72f9aaff93c4696bd1bdd1b78e0f280, State: Running, Role: LEADER
I20260128 15:02:11.865276 15949 consensus_queue.cc:237] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:11.867106 15967 tablet_replica.cc:442] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280: TxnStatusTablet state changed. Reason: New leader a72f9aaff93c4696bd1bdd1b78e0f280. Latest consensus state: current_term: 1 leader_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } } }
I20260128 15:02:11.867205 15967 tablet_replica.cc:445] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280: This TxnStatusTablet replica's current role is: LEADER
I20260128 15:02:11.867467 15969 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260128 15:02:11.867655 15969 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260128 15:02:11.868255 15429 catalog_manager.cc:5654] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 reported cstate change: term changed from 0 to 1, leader changed from <none> to a72f9aaff93c4696bd1bdd1b78e0f280 (127.13.88.130). New cstate: current_term: 1 leader_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } health_report { overall_health: UNKNOWN } } }
I20260128 15:02:11.903575 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.900788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.13.88.130:40711)
0128 15:02:11.900912 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.900916 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.900930 (+ 14us) server_negotiation.cc:408] Connection header received
0128 15:02:11.901085 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.901088 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.901141 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.901213 (+ 72us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:02:11.901527 (+ 314us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.901973 (+ 446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.902571 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.902749 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.902796 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:02:11.903205 (+ 409us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:02:11.903300 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.903390 (+ 90us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.903438 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260128 15:02:11.903575 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.900688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:40711 (local address 127.0.0.1:57936)
0128 15:02:11.900836 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:11.900852 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:11.900963 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:11.901233 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:11.901235 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:11.901244 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 15:02:11.901428 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.901434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.902070 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.902074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:11.902474 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.902481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.902576 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.902622 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 15:02:11.903297 (+ 675us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 15:02:11.903313 (+ 16us) client_negotiation.cc:769] Sending connection context
0128 15:02:11.903373 (+ 60us) client_negotiation.cc:240] Negotiation successful
0128 15:02:11.903452 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260128 15:02:11.908027 15873 raft_consensus.cc:1275] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Refusing update from remote peer a72f9aaff93c4696bd1bdd1b78e0f280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:11.908025 15589 raft_consensus.cc:1275] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865 [term 1 FOLLOWER]: Refusing update from remote peer a72f9aaff93c4696bd1bdd1b78e0f280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:11.908651 15949 consensus_queue.cc:1048] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:11.908805 15967 consensus_queue.cc:1048] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:11.911537 15948 tablet_replica.cc:442] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865: TxnStatusTablet state changed. Reason: New leader a72f9aaff93c4696bd1bdd1b78e0f280. Latest consensus state: current_term: 1 leader_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } } }
I20260128 15:02:11.911638 15948 tablet_replica.cc:445] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:02:11.912483 15950 tablet_replica.cc:442] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470: TxnStatusTablet state changed. Reason: New leader a72f9aaff93c4696bd1bdd1b78e0f280. Latest consensus state: current_term: 1 leader_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } } }
I20260128 15:02:11.912601 15950 tablet_replica.cc:445] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:02:11.915016 15949 tablet_replica.cc:442] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } } }
I20260128 15:02:11.915124 15949 tablet_replica.cc:445] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280: This TxnStatusTablet replica's current role is: LEADER
I20260128 15:02:11.915133 15948 tablet_replica.cc:442] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } } }
I20260128 15:02:11.915221 15948 tablet_replica.cc:445] T 9a0221fc44b94dabbb1e2237ef7a7b82 P f365650f1de74e8ca759210dba0d2865: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:02:11.915236 15949 tablet_replica.cc:442] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } } }
I20260128 15:02:11.915300 15949 tablet_replica.cc:445] T 9a0221fc44b94dabbb1e2237ef7a7b82 P a72f9aaff93c4696bd1bdd1b78e0f280: This TxnStatusTablet replica's current role is: LEADER
I20260128 15:02:11.915759 15972 mvcc.cc:204] Tried to move back new op lower bound from 7248332930689306624 to 7248332930522611712. Current Snapshot: MvccSnapshot[applied={T|T < 7248332930689306624}]
I20260128 15:02:11.917057 15950 tablet_replica.cc:442] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } } }
I20260128 15:02:11.917148 15950 tablet_replica.cc:445] T 9a0221fc44b94dabbb1e2237ef7a7b82 P fd1d86066bc449b2b6527f6056048470: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:02:11.917771 15974 mvcc.cc:204] Tried to move back new op lower bound from 7248332930689306624 to 7248332930522611712. Current Snapshot: MvccSnapshot[applied={T|T < 7248332930689306624}]
I20260128 15:02:11.924578 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.921316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.13.88.129:35019)
0128 15:02:11.921439 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.921443 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.921611 (+ 168us) server_negotiation.cc:408] Connection header received
0128 15:02:11.921765 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.921768 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.921814 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.921885 (+ 71us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:02:11.922291 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.922775 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.923654 (+ 879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.923817 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.923865 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:02:11.924231 (+ 366us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:02:11.924312 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.924396 (+ 84us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.924467 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260128 15:02:11.933941 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.931127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:37497 (local address 127.13.88.130:40711)
0128 15:02:11.931251 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.931254 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.931334 (+ 80us) server_negotiation.cc:408] Connection header received
0128 15:02:11.931453 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.931456 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.931508 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.931565 (+ 57us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:02:11.931972 (+ 407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.932533 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.933418 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.933560 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.933597 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:02:11.933660 (+ 63us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:02:11.933713 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.933811 (+ 98us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.933846 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260128 15:02:11.934006 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.931008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:40711 (local address 127.13.88.129:37497)
0128 15:02:11.931255 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:11.931275 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:11.931372 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:11.931585 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:11.931589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:11.931596 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 15:02:11.931885 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.931893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.932637 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.932641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:11.933324 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.933334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.933448 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.933494 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 15:02:11.933734 (+ 240us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 15:02:11.933739 (+ 5us) client_negotiation.cc:769] Sending connection context
0128 15:02:11.933794 (+ 55us) client_negotiation.cc:240] Negotiation successful
0128 15:02:11.933861 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260128 15:02:11.940155 15873 raft_consensus.cc:1275] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Refusing update from remote peer f365650f1de74e8ca759210dba0d2865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:11.940440 15731 raft_consensus.cc:1275] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 FOLLOWER]: Refusing update from remote peer f365650f1de74e8ca759210dba0d2865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:11.940569 15948 consensus_queue.cc:1048] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:11.940887 15948 consensus_queue.cc:1048] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:11.946326 15973 mvcc.cc:204] Tried to move back new op lower bound from 7248332930823397376 to 7248332929235865600. Current Snapshot: MvccSnapshot[applied={T|T < 7248332930823397376}]
I20260128 15:02:11.962878 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.959649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:37545 (local address 127.13.88.129:35019)
0128 15:02:11.959764 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:11.959767 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:11.959779 (+ 12us) server_negotiation.cc:408] Connection header received
0128 15:02:11.959902 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:11.959904 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:11.959943 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:11.960001 (+ 58us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:02:11.960955 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.961477 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.962342 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.962478 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.962523 (+ 45us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:02:11.962585 (+ 62us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:02:11.962640 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:11.962743 (+ 103us) server_negotiation.cc:300] Negotiation successful
0128 15:02:11.962782 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260128 15:02:11.962950 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:11.959403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:35019 (local address 127.13.88.130:37545)
0128 15:02:11.959716 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:11.959732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:11.959822 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:11.960021 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:11.960024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:11.960032 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 15:02:11.960801 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.960811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.961584 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:11.961588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:11.962241 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:11.962250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:11.962368 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:11.962412 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 15:02:11.962662 (+ 250us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 15:02:11.962669 (+ 7us) client_negotiation.cc:769] Sending connection context
0128 15:02:11.962723 (+ 54us) client_negotiation.cc:240] Negotiation successful
0128 15:02:11.962790 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":102,"threads_started":1}
I20260128 15:02:11.989144 15428 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:02:11.989336 15428 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:02:11.991899 15428 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb: Sending DeleteTablet for 3 replicas of tablet 2c55e926c50c4c96b0c451a78c052464
I20260128 15:02:11.992561 15569 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c55e926c50c4c96b0c451a78c052464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:11 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:42412
I20260128 15:02:11.992607 15711 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c55e926c50c4c96b0c451a78c052464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:11 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:41870
I20260128 15:02:11.993163 15992 tablet_replica.cc:333] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865: stopping tablet replica
I20260128 15:02:11.993376 15853 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c55e926c50c4c96b0c451a78c052464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:11 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:44792
I20260128 15:02:11.993451 15992 raft_consensus.cc:2243] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [term 1 LEADER]: Raft consensus shutting down.
I20260128 15:02:11.993798 15992 raft_consensus.cc:2272] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:11.994642 15429 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 15:02:11.995041 15429 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:02:11.995091 15992 ts_tablet_manager.cc:1916] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:11.997891 15992 ts_tablet_manager.cc:1929] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 15:02:11.998049 15992 log.cc:1199] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/wal/wals/2c55e926c50c4c96b0c451a78c052464
I20260128 15:02:11.998420 15992 ts_tablet_manager.cc:1950] T 2c55e926c50c4c96b0c451a78c052464 P f365650f1de74e8ca759210dba0d2865: Deleting consensus metadata
I20260128 15:02:11.999514 15416 catalog_manager.cc:4985] TS f365650f1de74e8ca759210dba0d2865 (127.13.88.129:35019): tablet 2c55e926c50c4c96b0c451a78c052464 (table test-table [id=096721b8bb3345f09cd6a631bc72361d]) successfully deleted
I20260128 15:02:12.000371 15569 tablet_service.cc:1505] Processing CreateTablet for tablet e8ffce389a1d4da0ba5ec53f6ededd58 (DEFAULT_TABLE table=test-table [id=3e8d3e18d7564a398509a28377224dca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:12.000638 15569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8ffce389a1d4da0ba5ec53f6ededd58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:12.000780 15711 tablet_service.cc:1505] Processing CreateTablet for tablet e8ffce389a1d4da0ba5ec53f6ededd58 (DEFAULT_TABLE table=test-table [id=3e8d3e18d7564a398509a28377224dca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:12.001016 15711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8ffce389a1d4da0ba5ec53f6ededd58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:12.002619 15942 tablet_bootstrap.cc:492] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865: Bootstrap starting.
I20260128 15:02:12.003101 15943 tablet_bootstrap.cc:492] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280: Bootstrap starting.
I20260128 15:02:12.003582 15942 tablet_bootstrap.cc:654] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:12.003764 15853 tablet_service.cc:1505] Processing CreateTablet for tablet e8ffce389a1d4da0ba5ec53f6ededd58 (DEFAULT_TABLE table=test-table [id=3e8d3e18d7564a398509a28377224dca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:12.003961 15943 tablet_bootstrap.cc:654] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:12.004000 15853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8ffce389a1d4da0ba5ec53f6ededd58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:12.005030 15942 tablet_bootstrap.cc:492] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865: No bootstrap required, opened a new log
I20260128 15:02:12.005110 15942 ts_tablet_manager.cc:1403] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 15:02:12.005537 15942 raft_consensus.cc:359] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:12.005632 15942 raft_consensus.cc:385] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:12.005661 15942 raft_consensus.cc:740] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f365650f1de74e8ca759210dba0d2865, State: Initialized, Role: FOLLOWER
I20260128 15:02:12.005772 15942 consensus_queue.cc:260] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:12.006011 15942 ts_tablet_manager.cc:1434] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:12.006323 15945 tablet_bootstrap.cc:492] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470: Bootstrap starting.
I20260128 15:02:12.007232 15945 tablet_bootstrap.cc:654] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:12.008240 15994 tablet_replica.cc:333] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470: stopping tablet replica
I20260128 15:02:12.008553 15994 raft_consensus.cc:2243] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:02:12.008805 15994 raft_consensus.cc:2272] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:12.009104 15945 tablet_bootstrap.cc:492] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470: No bootstrap required, opened a new log
I20260128 15:02:12.009183 15945 ts_tablet_manager.cc:1403] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260128 15:02:12.009191 15993 tablet_replica.cc:333] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280: stopping tablet replica
I20260128 15:02:12.009402 15993 raft_consensus.cc:2243] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:02:12.009545 15993 raft_consensus.cc:2272] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:12.009666 15945 raft_consensus.cc:359] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:12.009766 15945 raft_consensus.cc:385] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:12.009799 15945 raft_consensus.cc:740] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1d86066bc449b2b6527f6056048470, State: Initialized, Role: FOLLOWER
I20260128 15:02:12.009903 15945 consensus_queue.cc:260] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:12.010134 15945 ts_tablet_manager.cc:1434] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:12.010327 15994 ts_tablet_manager.cc:1916] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:12.013630 15993 ts_tablet_manager.cc:1916] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:12.016505 15993 ts_tablet_manager.cc:1929] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 15:02:12.016592 15993 log.cc:1199] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/wal/wals/2c55e926c50c4c96b0c451a78c052464
I20260128 15:02:12.016892 15993 ts_tablet_manager.cc:1950] T 2c55e926c50c4c96b0c451a78c052464 P a72f9aaff93c4696bd1bdd1b78e0f280: Deleting consensus metadata
I20260128 15:02:12.017961 15417 catalog_manager.cc:4985] TS a72f9aaff93c4696bd1bdd1b78e0f280 (127.13.88.130:40711): tablet 2c55e926c50c4c96b0c451a78c052464 (table test-table [id=096721b8bb3345f09cd6a631bc72361d]) successfully deleted
I20260128 15:02:12.021085 15943 tablet_bootstrap.cc:492] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280: No bootstrap required, opened a new log
I20260128 15:02:12.021185 15943 ts_tablet_manager.cc:1403] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.000s
I20260128 15:02:12.021685 15943 raft_consensus.cc:359] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:12.021799 15943 raft_consensus.cc:385] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:12.021829 15943 raft_consensus.cc:740] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72f9aaff93c4696bd1bdd1b78e0f280, State: Initialized, Role: FOLLOWER
I20260128 15:02:12.021955 15943 consensus_queue.cc:260] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:12.022195 15943 ts_tablet_manager.cc:1434] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260128 15:02:12.022373 15994 ts_tablet_manager.cc:1929] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 15:02:12.022450 15994 log.cc:1199] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/wal/wals/2c55e926c50c4c96b0c451a78c052464
I20260128 15:02:12.022790 15994 ts_tablet_manager.cc:1950] T 2c55e926c50c4c96b0c451a78c052464 P fd1d86066bc449b2b6527f6056048470: Deleting consensus metadata
I20260128 15:02:12.024787 15415 catalog_manager.cc:4985] TS fd1d86066bc449b2b6527f6056048470 (127.13.88.131:42773): tablet 2c55e926c50c4c96b0c451a78c052464 (table test-table [id=096721b8bb3345f09cd6a631bc72361d]) successfully deleted
I20260128 15:02:12.103281 15967 raft_consensus.cc:493] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:02:12.103441 15967 raft_consensus.cc:515] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:12.103926 15967 leader_election.cc:290] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f365650f1de74e8ca759210dba0d2865 (127.13.88.129:35019), fd1d86066bc449b2b6527f6056048470 (127.13.88.131:42773)
I20260128 15:02:12.104331 15589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8ffce389a1d4da0ba5ec53f6ededd58" candidate_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f365650f1de74e8ca759210dba0d2865" is_pre_election: true
I20260128 15:02:12.104362 15872 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8ffce389a1d4da0ba5ec53f6ededd58" candidate_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd1d86066bc449b2b6527f6056048470" is_pre_election: true
I20260128 15:02:12.104490 15589 raft_consensus.cc:2468] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72f9aaff93c4696bd1bdd1b78e0f280 in term 0.
I20260128 15:02:12.104524 15872 raft_consensus.cc:2468] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72f9aaff93c4696bd1bdd1b78e0f280 in term 0.
I20260128 15:02:12.104795 15664 leader_election.cc:304] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72f9aaff93c4696bd1bdd1b78e0f280, f365650f1de74e8ca759210dba0d2865; no voters:
I20260128 15:02:12.104995 15967 raft_consensus.cc:2804] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:02:12.105059 15967 raft_consensus.cc:493] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:02:12.105093 15967 raft_consensus.cc:3060] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:12.106238 15967 raft_consensus.cc:515] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:12.106576 15967 leader_election.cc:290] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [CANDIDATE]: Term 1 election: Requested vote from peers f365650f1de74e8ca759210dba0d2865 (127.13.88.129:35019), fd1d86066bc449b2b6527f6056048470 (127.13.88.131:42773)
I20260128 15:02:12.106871 15589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8ffce389a1d4da0ba5ec53f6ededd58" candidate_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f365650f1de74e8ca759210dba0d2865"
I20260128 15:02:12.106904 15872 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8ffce389a1d4da0ba5ec53f6ededd58" candidate_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd1d86066bc449b2b6527f6056048470"
I20260128 15:02:12.106978 15589 raft_consensus.cc:3060] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:12.106998 15872 raft_consensus.cc:3060] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:12.107920 15589 raft_consensus.cc:2468] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72f9aaff93c4696bd1bdd1b78e0f280 in term 1.
I20260128 15:02:12.107929 15872 raft_consensus.cc:2468] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72f9aaff93c4696bd1bdd1b78e0f280 in term 1.
I20260128 15:02:12.108191 15663 leader_election.cc:304] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72f9aaff93c4696bd1bdd1b78e0f280, fd1d86066bc449b2b6527f6056048470; no voters:
I20260128 15:02:12.108505 15967 raft_consensus.cc:2804] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:12.108584 15967 raft_consensus.cc:697] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 LEADER]: Becoming Leader. State: Replica: a72f9aaff93c4696bd1bdd1b78e0f280, State: Running, Role: LEADER
I20260128 15:02:12.108716 15967 consensus_queue.cc:237] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:12.110363 15429 catalog_manager.cc:5654] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 reported cstate change: term changed from 0 to 1, leader changed from <none> to a72f9aaff93c4696bd1bdd1b78e0f280 (127.13.88.130). New cstate: current_term: 1 leader_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } health_report { overall_health: UNKNOWN } } }
I20260128 15:02:12.144785 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:12.142089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.13.88.130:40711)
0128 15:02:12.142218 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:12.142222 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:12.142236 (+ 14us) server_negotiation.cc:408] Connection header received
0128 15:02:12.142278 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:12.142281 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:12.142335 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:12.142417 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:02:12.142732 (+ 315us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:12.143474 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:12.144066 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:12.144253 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:12.144306 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:02:12.144387 (+ 81us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:02:12.144489 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:12.144580 (+ 91us) server_negotiation.cc:300] Negotiation successful
0128 15:02:12.144635 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260128 15:02:12.153402 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:12.150525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:40711 (local address 127.13.88.130:39207)
0128 15:02:12.150720 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:12.150732 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:12.150812 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:12.151046 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:12.151049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:12.151057 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 15:02:12.151222 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:12.151227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:12.152154 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:12.152158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:12.152588 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:12.152596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:12.152691 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:12.152714 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 15:02:12.153137 (+ 423us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 15:02:12.153143 (+ 6us) client_negotiation.cc:769] Sending connection context
0128 15:02:12.153209 (+ 66us) client_negotiation.cc:240] Negotiation successful
0128 15:02:12.153267 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260128 15:02:12.153422 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:12.150596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:39207 (local address 127.13.88.130:40711)
0128 15:02:12.150734 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:12.150739 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:12.150771 (+ 32us) server_negotiation.cc:408] Connection header received
0128 15:02:12.150894 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:12.150898 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:12.150951 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:12.151025 (+ 74us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:02:12.151312 (+ 287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:12.152053 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:12.152703 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:12.152889 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:12.152937 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:02:12.153011 (+ 74us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:02:12.153094 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:12.153238 (+ 144us) server_negotiation.cc:300] Negotiation successful
0128 15:02:12.153287 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260128 15:02:12.159442 15872 raft_consensus.cc:1275] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Refusing update from remote peer a72f9aaff93c4696bd1bdd1b78e0f280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:12.159482 15589 raft_consensus.cc:1275] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865 [term 1 FOLLOWER]: Refusing update from remote peer a72f9aaff93c4696bd1bdd1b78e0f280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:12.159802 15967 consensus_queue.cc:1048] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:12.160023 15949 consensus_queue.cc:1048] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260128 15:02:12.174521 15972 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20260128 15:02:13.727358 15428 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:02:13.727494 15428 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:02:13.729645 15428 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb: Sending DeleteTablet for 3 replicas of tablet e8ffce389a1d4da0ba5ec53f6ededd58
I20260128 15:02:13.730111 15569 tablet_service.cc:1552] Processing DeleteTablet for tablet e8ffce389a1d4da0ba5ec53f6ededd58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:13 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:42412
I20260128 15:02:13.730140 15711 tablet_service.cc:1552] Processing DeleteTablet for tablet e8ffce389a1d4da0ba5ec53f6ededd58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:13 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:41870
I20260128 15:02:13.730218 15853 tablet_service.cc:1552] Processing DeleteTablet for tablet e8ffce389a1d4da0ba5ec53f6ededd58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:13 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:44792
I20260128 15:02:13.730439 16022 tablet_replica.cc:333] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865: stopping tablet replica
I20260128 15:02:13.730454 16023 tablet_replica.cc:333] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280: stopping tablet replica
I20260128 15:02:13.730566 16023 raft_consensus.cc:2243] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 LEADER]: Raft consensus shutting down.
I20260128 15:02:13.730609 16024 tablet_replica.cc:333] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470: stopping tablet replica
I20260128 15:02:13.730605 16022 raft_consensus.cc:2243] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:02:13.730718 16022 raft_consensus.cc:2272] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:13.730762 16024 raft_consensus.cc:2243] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:02:13.730938 16024 raft_consensus.cc:2272] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:13.731155 16023 raft_consensus.cc:2272] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:13.731334 16022 ts_tablet_manager.cc:1916] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:13.731668 16023 ts_tablet_manager.cc:1916] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:13.733384 16024 ts_tablet_manager.cc:1916] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:13.733996 16022 ts_tablet_manager.cc:1929] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260128 15:02:13.734102 16022 log.cc:1199] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/wal/wals/e8ffce389a1d4da0ba5ec53f6ededd58
I20260128 15:02:13.734154 16023 ts_tablet_manager.cc:1929] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260128 15:02:13.734261 16023 log.cc:1199] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/wal/wals/e8ffce389a1d4da0ba5ec53f6ededd58
I20260128 15:02:13.734393 16022 ts_tablet_manager.cc:1950] T e8ffce389a1d4da0ba5ec53f6ededd58 P f365650f1de74e8ca759210dba0d2865: Deleting consensus metadata
I20260128 15:02:13.734534 16023 ts_tablet_manager.cc:1950] T e8ffce389a1d4da0ba5ec53f6ededd58 P a72f9aaff93c4696bd1bdd1b78e0f280: Deleting consensus metadata
I20260128 15:02:13.735002 15416 catalog_manager.cc:4985] TS f365650f1de74e8ca759210dba0d2865 (127.13.88.129:35019): tablet e8ffce389a1d4da0ba5ec53f6ededd58 (table test-table [id=3e8d3e18d7564a398509a28377224dca]) successfully deleted
I20260128 15:02:13.735179 15417 catalog_manager.cc:4985] TS a72f9aaff93c4696bd1bdd1b78e0f280 (127.13.88.130:40711): tablet e8ffce389a1d4da0ba5ec53f6ededd58 (table test-table [id=3e8d3e18d7564a398509a28377224dca]) successfully deleted
I20260128 15:02:13.735767 16024 ts_tablet_manager.cc:1929] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260128 15:02:13.735831 16024 log.cc:1199] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/wal/wals/e8ffce389a1d4da0ba5ec53f6ededd58
I20260128 15:02:13.736016 16024 ts_tablet_manager.cc:1950] T e8ffce389a1d4da0ba5ec53f6ededd58 P fd1d86066bc449b2b6527f6056048470: Deleting consensus metadata
I20260128 15:02:13.736545 15415 catalog_manager.cc:4985] TS fd1d86066bc449b2b6527f6056048470 (127.13.88.131:42773): tablet e8ffce389a1d4da0ba5ec53f6ededd58 (table test-table [id=3e8d3e18d7564a398509a28377224dca]) successfully deleted
Jan 28 15:02:13 dist-test-slave-m28t krb5kdc[15382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612533, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 28 15:02:13 dist-test-slave-m28t krb5kdc[15382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612533, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:13.760846 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:13.749386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.13.88.190:33355)
0128 15:02:13.749652 (+ 266us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:13.749656 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:13.749671 (+ 15us) server_negotiation.cc:408] Connection header received
0128 15:02:13.749754 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:13.749758 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:13.749804 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:13.749878 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:13.750651 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.751170 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:13.751804 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.751990 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:13.758322 (+ 6332us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:13.758341 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:13.758343 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:13.758370 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:13.759608 (+ 1238us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:13.759978 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:13.759981 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:13.759982 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:13.760021 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:13.760163 (+ 142us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:13.760165 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:13.760166 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:13.760306 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:13.760398 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:13.760605 (+ 207us) server_negotiation.cc:300] Negotiation successful
0128 15:02:13.760709 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
I20260128 15:02:13.763096 15428 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 15:02:13.763513 15428 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:02:13.767953 15569 tablet_service.cc:1505] Processing CreateTablet for tablet c859982bc14846bfb621d78d36c47fcb (DEFAULT_TABLE table=test-table [id=dc8fb4a6f29643b09822ba33ed664e58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:13.768085 15711 tablet_service.cc:1505] Processing CreateTablet for tablet c859982bc14846bfb621d78d36c47fcb (DEFAULT_TABLE table=test-table [id=dc8fb4a6f29643b09822ba33ed664e58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:13.768278 15569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c859982bc14846bfb621d78d36c47fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:13.768364 15711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c859982bc14846bfb621d78d36c47fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:13.768352 15853 tablet_service.cc:1505] Processing CreateTablet for tablet c859982bc14846bfb621d78d36c47fcb (DEFAULT_TABLE table=test-table [id=dc8fb4a6f29643b09822ba33ed664e58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:13.768635 15853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c859982bc14846bfb621d78d36c47fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:13.770224 16033 tablet_bootstrap.cc:492] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280: Bootstrap starting.
I20260128 15:02:13.770903 16035 tablet_bootstrap.cc:492] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470: Bootstrap starting.
I20260128 15:02:13.771000 16034 tablet_bootstrap.cc:492] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865: Bootstrap starting.
I20260128 15:02:13.771124 16033 tablet_bootstrap.cc:654] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:13.771754 16035 tablet_bootstrap.cc:654] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:13.771809 16034 tablet_bootstrap.cc:654] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:13.772204 16033 tablet_bootstrap.cc:492] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280: No bootstrap required, opened a new log
I20260128 15:02:13.772332 16033 ts_tablet_manager.cc:1403] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:13.772812 16033 raft_consensus.cc:359] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.772928 16033 raft_consensus.cc:385] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:13.772960 16033 raft_consensus.cc:740] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72f9aaff93c4696bd1bdd1b78e0f280, State: Initialized, Role: FOLLOWER
I20260128 15:02:13.772997 16035 tablet_bootstrap.cc:492] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470: No bootstrap required, opened a new log
I20260128 15:02:13.773092 16035 ts_tablet_manager.cc:1403] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:13.773110 16033 consensus_queue.cc:260] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.773375 16033 ts_tablet_manager.cc:1434] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:13.773487 16035 raft_consensus.cc:359] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.773571 16035 raft_consensus.cc:385] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:13.773591 16035 raft_consensus.cc:740] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1d86066bc449b2b6527f6056048470, State: Initialized, Role: FOLLOWER
I20260128 15:02:13.773696 16035 consensus_queue.cc:260] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.774041 16035 ts_tablet_manager.cc:1434] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:13.774776 16034 tablet_bootstrap.cc:492] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865: No bootstrap required, opened a new log
I20260128 15:02:13.774865 16034 ts_tablet_manager.cc:1403] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260128 15:02:13.775282 16034 raft_consensus.cc:359] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.775388 16034 raft_consensus.cc:385] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:13.775485 16034 raft_consensus.cc:740] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f365650f1de74e8ca759210dba0d2865, State: Initialized, Role: FOLLOWER
I20260128 15:02:13.775687 16034 consensus_queue.cc:260] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.776072 16034 ts_tablet_manager.cc:1434] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:13.800282 15949 raft_consensus.cc:493] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:02:13.800482 15949 raft_consensus.cc:515] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.800894 15949 leader_election.cc:290] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f365650f1de74e8ca759210dba0d2865 (127.13.88.129:35019), fd1d86066bc449b2b6527f6056048470 (127.13.88.131:42773)
I20260128 15:02:13.801235 15589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c859982bc14846bfb621d78d36c47fcb" candidate_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f365650f1de74e8ca759210dba0d2865" is_pre_election: true
I20260128 15:02:13.801277 15873 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c859982bc14846bfb621d78d36c47fcb" candidate_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd1d86066bc449b2b6527f6056048470" is_pre_election: true
I20260128 15:02:13.801373 15589 raft_consensus.cc:2468] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72f9aaff93c4696bd1bdd1b78e0f280 in term 0.
I20260128 15:02:13.801395 15873 raft_consensus.cc:2468] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a72f9aaff93c4696bd1bdd1b78e0f280 in term 0.
I20260128 15:02:13.801702 15663 leader_election.cc:304] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a72f9aaff93c4696bd1bdd1b78e0f280, f365650f1de74e8ca759210dba0d2865, fd1d86066bc449b2b6527f6056048470; no voters:
I20260128 15:02:13.801882 15949 raft_consensus.cc:2804] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:02:13.801937 15949 raft_consensus.cc:493] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:02:13.801973 15949 raft_consensus.cc:3060] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:13.802822 15949 raft_consensus.cc:515] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.803125 15949 leader_election.cc:290] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [CANDIDATE]: Term 1 election: Requested vote from peers f365650f1de74e8ca759210dba0d2865 (127.13.88.129:35019), fd1d86066bc449b2b6527f6056048470 (127.13.88.131:42773)
I20260128 15:02:13.803474 15873 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c859982bc14846bfb621d78d36c47fcb" candidate_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd1d86066bc449b2b6527f6056048470"
I20260128 15:02:13.803516 15589 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c859982bc14846bfb621d78d36c47fcb" candidate_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f365650f1de74e8ca759210dba0d2865"
I20260128 15:02:13.803591 15873 raft_consensus.cc:3060] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:13.803612 15589 raft_consensus.cc:3060] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:13.804391 15589 raft_consensus.cc:2468] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72f9aaff93c4696bd1bdd1b78e0f280 in term 1.
I20260128 15:02:13.804396 15873 raft_consensus.cc:2468] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a72f9aaff93c4696bd1bdd1b78e0f280 in term 1.
I20260128 15:02:13.804694 15663 leader_election.cc:304] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72f9aaff93c4696bd1bdd1b78e0f280, fd1d86066bc449b2b6527f6056048470; no voters:
I20260128 15:02:13.804853 15949 raft_consensus.cc:2804] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:13.804940 15949 raft_consensus.cc:697] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 LEADER]: Becoming Leader. State: Replica: a72f9aaff93c4696bd1bdd1b78e0f280, State: Running, Role: LEADER
I20260128 15:02:13.805066 15949 consensus_queue.cc:237] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.806576 15428 catalog_manager.cc:5654] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 reported cstate change: term changed from 0 to 1, leader changed from <none> to a72f9aaff93c4696bd1bdd1b78e0f280 (127.13.88.130). New cstate: current_term: 1 leader_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } health_report { overall_health: UNKNOWN } } }
I20260128 15:02:13.818670 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:13.816290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.13.88.130:40711)
0128 15:02:13.816602 (+ 312us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:13.816605 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:13.816617 (+ 12us) server_negotiation.cc:408] Connection header received
0128 15:02:13.816651 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:13.816653 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:13.816694 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:13.816762 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:02:13.817067 (+ 305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.817569 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:13.818125 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.818262 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:13.818299 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:02:13.818361 (+ 62us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:02:13.818419 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:13.818493 (+ 74us) server_negotiation.cc:300] Negotiation successful
0128 15:02:13.818534 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":101,"threads_started":1}
I20260128 15:02:13.823848 15873 raft_consensus.cc:1275] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Refusing update from remote peer a72f9aaff93c4696bd1bdd1b78e0f280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:13.823951 15588 raft_consensus.cc:1275] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865 [term 1 FOLLOWER]: Refusing update from remote peer a72f9aaff93c4696bd1bdd1b78e0f280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:13.824242 16005 consensus_queue.cc:1048] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:13.824443 16010 consensus_queue.cc:1048] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:13.853610 15430 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:02:13.853773 15430 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:02:13.855669 15430 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb: Sending DeleteTablet for 3 replicas of tablet c859982bc14846bfb621d78d36c47fcb
I20260128 15:02:13.856209 15710 tablet_service.cc:1552] Processing DeleteTablet for tablet c859982bc14846bfb621d78d36c47fcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:13 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:41870
I20260128 15:02:13.856232 15853 tablet_service.cc:1552] Processing DeleteTablet for tablet c859982bc14846bfb621d78d36c47fcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:13 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:44792
I20260128 15:02:13.856370 16023 tablet_replica.cc:333] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280: stopping tablet replica
I20260128 15:02:13.856405 16024 tablet_replica.cc:333] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470: stopping tablet replica
I20260128 15:02:13.856482 16023 raft_consensus.cc:2243] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 LEADER]: Raft consensus shutting down.
I20260128 15:02:13.856531 16024 raft_consensus.cc:2243] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:02:13.856639 16023 raft_consensus.cc:2272] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:13.856637 16024 raft_consensus.cc:2272] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:13.857038 15430 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260128 15:02:13.857403 16023 ts_tablet_manager.cc:1916] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260128 15:02:13.857411 15430 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:02:13.858870 16024 ts_tablet_manager.cc:1916] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:13.859628 16023 ts_tablet_manager.cc:1929] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 15:02:13.859704 16023 log.cc:1199] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/wal/wals/c859982bc14846bfb621d78d36c47fcb
I20260128 15:02:13.859900 16023 ts_tablet_manager.cc:1950] T c859982bc14846bfb621d78d36c47fcb P a72f9aaff93c4696bd1bdd1b78e0f280: Deleting consensus metadata
I20260128 15:02:13.860601 15417 catalog_manager.cc:4985] TS a72f9aaff93c4696bd1bdd1b78e0f280 (127.13.88.130:40711): tablet c859982bc14846bfb621d78d36c47fcb (table test-table [id=dc8fb4a6f29643b09822ba33ed664e58]) successfully deleted
I20260128 15:02:13.860905 15569 tablet_service.cc:1552] Processing DeleteTablet for tablet c859982bc14846bfb621d78d36c47fcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:13 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:42412
I20260128 15:02:13.861011 16024 ts_tablet_manager.cc:1929] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 15:02:13.861066 16024 log.cc:1199] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/wal/wals/c859982bc14846bfb621d78d36c47fcb
I20260128 15:02:13.861073 16022 tablet_replica.cc:333] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865: stopping tablet replica
I20260128 15:02:13.861177 16022 raft_consensus.cc:2243] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:02:13.861268 16024 ts_tablet_manager.cc:1950] T c859982bc14846bfb621d78d36c47fcb P fd1d86066bc449b2b6527f6056048470: Deleting consensus metadata
I20260128 15:02:13.861279 16022 raft_consensus.cc:2272] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:13.861865 15415 catalog_manager.cc:4985] TS fd1d86066bc449b2b6527f6056048470 (127.13.88.131:42773): tablet c859982bc14846bfb621d78d36c47fcb (table test-table [id=dc8fb4a6f29643b09822ba33ed664e58]) successfully deleted
I20260128 15:02:13.862262 16022 ts_tablet_manager.cc:1916] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:13.863483 15569 tablet_service.cc:1505] Processing CreateTablet for tablet cc8ce026376c47bfbffc0c33f0d3770a (DEFAULT_TABLE table=test-table [id=b3009b16213c4c67b42c93c9a2f5c17c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:13.863745 15569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc8ce026376c47bfbffc0c33f0d3770a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:13.863950 15853 tablet_service.cc:1505] Processing CreateTablet for tablet cc8ce026376c47bfbffc0c33f0d3770a (DEFAULT_TABLE table=test-table [id=b3009b16213c4c67b42c93c9a2f5c17c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:13.864194 15853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc8ce026376c47bfbffc0c33f0d3770a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:13.864404 16022 ts_tablet_manager.cc:1929] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 15:02:13.864483 16022 log.cc:1199] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-0/wal/wals/c859982bc14846bfb621d78d36c47fcb
I20260128 15:02:13.864720 16022 ts_tablet_manager.cc:1950] T c859982bc14846bfb621d78d36c47fcb P f365650f1de74e8ca759210dba0d2865: Deleting consensus metadata
I20260128 15:02:13.865718 16034 tablet_bootstrap.cc:492] T cc8ce026376c47bfbffc0c33f0d3770a P f365650f1de74e8ca759210dba0d2865: Bootstrap starting.
I20260128 15:02:13.865890 15416 catalog_manager.cc:4985] TS f365650f1de74e8ca759210dba0d2865 (127.13.88.129:35019): tablet c859982bc14846bfb621d78d36c47fcb (table test-table [id=dc8fb4a6f29643b09822ba33ed664e58]) successfully deleted
I20260128 15:02:13.866197 16035 tablet_bootstrap.cc:492] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470: Bootstrap starting.
I20260128 15:02:13.866495 15710 tablet_service.cc:1505] Processing CreateTablet for tablet cc8ce026376c47bfbffc0c33f0d3770a (DEFAULT_TABLE table=test-table [id=b3009b16213c4c67b42c93c9a2f5c17c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:13.866709 15710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc8ce026376c47bfbffc0c33f0d3770a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:13.867041 16035 tablet_bootstrap.cc:654] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:13.868103 16035 tablet_bootstrap.cc:492] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470: No bootstrap required, opened a new log
I20260128 15:02:13.868170 16035 ts_tablet_manager.cc:1403] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:13.868454 16033 tablet_bootstrap.cc:492] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280: Bootstrap starting.
I20260128 15:02:13.868686 16035 raft_consensus.cc:359] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.868778 16035 raft_consensus.cc:385] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:13.868805 16035 raft_consensus.cc:740] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1d86066bc449b2b6527f6056048470, State: Initialized, Role: FOLLOWER
I20260128 15:02:13.868906 16035 consensus_queue.cc:260] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.869127 16035 ts_tablet_manager.cc:1434] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:13.869275 16033 tablet_bootstrap.cc:654] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:13.869814 16034 tablet_bootstrap.cc:654] T cc8ce026376c47bfbffc0c33f0d3770a P f365650f1de74e8ca759210dba0d2865: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:13.870486 16033 tablet_bootstrap.cc:492] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280: No bootstrap required, opened a new log
I20260128 15:02:13.870592 16033 ts_tablet_manager.cc:1403] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:13.871133 16033 raft_consensus.cc:359] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.871239 16033 raft_consensus.cc:385] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:13.871270 16033 raft_consensus.cc:740] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72f9aaff93c4696bd1bdd1b78e0f280, State: Initialized, Role: FOLLOWER
I20260128 15:02:13.871389 16033 consensus_queue.cc:260] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.871627 16033 ts_tablet_manager.cc:1434] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:13.873176 16034 tablet_bootstrap.cc:492] T cc8ce026376c47bfbffc0c33f0d3770a P f365650f1de74e8ca759210dba0d2865: No bootstrap required, opened a new log
I20260128 15:02:13.873246 16034 ts_tablet_manager.cc:1403] T cc8ce026376c47bfbffc0c33f0d3770a P f365650f1de74e8ca759210dba0d2865: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260128 15:02:13.873648 16034 raft_consensus.cc:359] T cc8ce026376c47bfbffc0c33f0d3770a P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.873754 16034 raft_consensus.cc:385] T cc8ce026376c47bfbffc0c33f0d3770a P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:13.873796 16034 raft_consensus.cc:740] T cc8ce026376c47bfbffc0c33f0d3770a P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f365650f1de74e8ca759210dba0d2865, State: Initialized, Role: FOLLOWER
I20260128 15:02:13.873924 16034 consensus_queue.cc:260] T cc8ce026376c47bfbffc0c33f0d3770a P f365650f1de74e8ca759210dba0d2865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.874177 16034 ts_tablet_manager.cc:1434] T cc8ce026376c47bfbffc0c33f0d3770a P f365650f1de74e8ca759210dba0d2865: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260128 15:02:13.943572 16039 raft_consensus.cc:493] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:02:13.943714 16039 raft_consensus.cc:515] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.944850 16039 leader_election.cc:290] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f365650f1de74e8ca759210dba0d2865 (127.13.88.129:35019), a72f9aaff93c4696bd1bdd1b78e0f280 (127.13.88.130:40711)
I20260128 15:02:13.948221 16051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:13.945054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:35019 (local address 127.13.88.131:36517)
0128 15:02:13.945387 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:13.945402 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:13.945514 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:13.945770 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:13.945773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:13.945823 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:13.946056 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:13.946061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.946832 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:13.946835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:13.947780 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:13.947789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.947899 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:13.947922 (+ 23us) client_negotiation.cc:769] Sending connection context
0128 15:02:13.947971 (+ 49us) client_negotiation.cc:240] Negotiation successful
0128 15:02:13.948030 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"spinlock_wait_cycles":22656,"thread_start_us":103,"threads_started":1}
I20260128 15:02:13.948221 16052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:13.945095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:40711 (local address 127.13.88.131:48691)
0128 15:02:13.945406 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:13.945420 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:13.945529 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:13.945771 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:13.945775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:13.945826 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:13.946054 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:13.946060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.946832 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:13.946835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:13.947780 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:13.947789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.947913 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:13.947924 (+ 11us) client_negotiation.cc:769] Sending connection context
0128 15:02:13.947971 (+ 47us) client_negotiation.cc:240] Negotiation successful
0128 15:02:13.948030 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"spinlock_wait_cycles":11392,"thread_start_us":114,"threads_started":1}
I20260128 15:02:13.948536 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:13.945323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:48691 (local address 127.13.88.130:40711)
0128 15:02:13.945435 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:13.945439 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:13.945502 (+ 63us) server_negotiation.cc:408] Connection header received
0128 15:02:13.945629 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:13.945632 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:13.945676 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:13.945739 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:13.946166 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.946727 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:13.947887 (+ 1160us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.948219 (+ 332us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:13.948253 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:13.948298 (+ 45us) server_negotiation.cc:300] Negotiation successful
0128 15:02:13.948354 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260128 15:02:13.948902 15731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc8ce026376c47bfbffc0c33f0d3770a" candidate_uuid: "fd1d86066bc449b2b6527f6056048470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" is_pre_election: true
I20260128 15:02:13.949050 15731 raft_consensus.cc:2468] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd1d86066bc449b2b6527f6056048470 in term 0.
I20260128 15:02:13.949120 16053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:13.945149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:36517 (local address 127.13.88.129:35019)
0128 15:02:13.945460 (+ 311us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:13.945465 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:13.945480 (+ 15us) server_negotiation.cc:408] Connection header received
0128 15:02:13.945627 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:13.945631 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:13.945674 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:13.945744 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:13.946170 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.946703 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:13.948283 (+ 1580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.948824 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:13.948858 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:13.948929 (+ 71us) server_negotiation.cc:300] Negotiation successful
0128 15:02:13.948981 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":70,"threads_started":1}
I20260128 15:02:13.949385 15807 leader_election.cc:304] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a72f9aaff93c4696bd1bdd1b78e0f280, fd1d86066bc449b2b6527f6056048470; no voters:
I20260128 15:02:13.949461 15588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc8ce026376c47bfbffc0c33f0d3770a" candidate_uuid: "fd1d86066bc449b2b6527f6056048470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f365650f1de74e8ca759210dba0d2865" is_pre_election: true
I20260128 15:02:13.949604 15588 raft_consensus.cc:2468] T cc8ce026376c47bfbffc0c33f0d3770a P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd1d86066bc449b2b6527f6056048470 in term 0.
I20260128 15:02:13.949622 16039 raft_consensus.cc:2804] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:02:13.949669 16039 raft_consensus.cc:493] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:02:13.949703 16039 raft_consensus.cc:3060] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:13.950721 16039 raft_consensus.cc:515] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.951110 16039 leader_election.cc:290] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [CANDIDATE]: Term 1 election: Requested vote from peers f365650f1de74e8ca759210dba0d2865 (127.13.88.129:35019), a72f9aaff93c4696bd1bdd1b78e0f280 (127.13.88.130:40711)
I20260128 15:02:13.951406 15588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc8ce026376c47bfbffc0c33f0d3770a" candidate_uuid: "fd1d86066bc449b2b6527f6056048470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f365650f1de74e8ca759210dba0d2865"
I20260128 15:02:13.951511 15588 raft_consensus.cc:3060] T cc8ce026376c47bfbffc0c33f0d3770a P f365650f1de74e8ca759210dba0d2865 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:13.952265 15588 raft_consensus.cc:2468] T cc8ce026376c47bfbffc0c33f0d3770a P f365650f1de74e8ca759210dba0d2865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd1d86066bc449b2b6527f6056048470 in term 1.
I20260128 15:02:13.952517 15731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc8ce026376c47bfbffc0c33f0d3770a" candidate_uuid: "fd1d86066bc449b2b6527f6056048470" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280"
I20260128 15:02:13.952538 15806 leader_election.cc:304] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f365650f1de74e8ca759210dba0d2865, fd1d86066bc449b2b6527f6056048470; no voters:
I20260128 15:02:13.952636 15731 raft_consensus.cc:3060] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:13.952698 16039 raft_consensus.cc:2804] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:13.952888 16039 raft_consensus.cc:697] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [term 1 LEADER]: Becoming Leader. State: Replica: fd1d86066bc449b2b6527f6056048470, State: Running, Role: LEADER
I20260128 15:02:13.953126 16039 consensus_queue.cc:237] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } }
I20260128 15:02:13.953653 15731 raft_consensus.cc:2468] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd1d86066bc449b2b6527f6056048470 in term 1.
I20260128 15:02:13.955353 15428 catalog_manager.cc:5654] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd1d86066bc449b2b6527f6056048470 (127.13.88.131). New cstate: current_term: 1 leader_uuid: "fd1d86066bc449b2b6527f6056048470" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd1d86066bc449b2b6527f6056048470" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 42773 } health_report { overall_health: HEALTHY } } }
I20260128 15:02:13.976909 16056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:13.974010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36860 (local address 127.13.88.131:42773)
0128 15:02:13.974260 (+ 250us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:13.974263 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:13.974274 (+ 11us) server_negotiation.cc:408] Connection header received
0128 15:02:13.974341 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:13.974343 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:13.974390 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:13.974482 (+ 92us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:02:13.974787 (+ 305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.975304 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:13.975865 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.975999 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:13.976048 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:02:13.976564 (+ 516us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:02:13.976653 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:13.976707 (+ 54us) server_negotiation.cc:300] Negotiation successful
0128 15:02:13.976758 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":114,"threads_started":1}
I20260128 15:02:13.985694 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:13.982841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:32931 (local address 127.13.88.130:40711)
0128 15:02:13.982974 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:13.982978 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:13.983089 (+ 111us) server_negotiation.cc:408] Connection header received
0128 15:02:13.983206 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:13.983209 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:13.983255 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:13.983315 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:02:13.983969 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.984444 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:13.985082 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.985257 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:13.985297 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:02:13.985358 (+ 61us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:02:13.985417 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:13.985529 (+ 112us) server_negotiation.cc:300] Negotiation successful
0128 15:02:13.985569 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260128 15:02:13.985710 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:13.982760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:40711 (local address 127.13.88.131:32931)
0128 15:02:13.983014 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:13.983026 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:13.983118 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:13.983330 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:13.983333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:13.983341 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 15:02:13.983858 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:13.983865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.984550 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:13.984553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:13.984974 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:13.984981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:13.985077 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:13.985116 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 15:02:13.985446 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 15:02:13.985451 (+ 5us) client_negotiation.cc:769] Sending connection context
0128 15:02:13.985514 (+ 63us) client_negotiation.cc:240] Negotiation successful
0128 15:02:13.985583 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
I20260128 15:02:13.991019 15731 raft_consensus.cc:1275] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 FOLLOWER]: Refusing update from remote peer fd1d86066bc449b2b6527f6056048470: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:13.991325 15588 raft_consensus.cc:1275] T cc8ce026376c47bfbffc0c33f0d3770a P f365650f1de74e8ca759210dba0d2865 [term 1 FOLLOWER]: Refusing update from remote peer fd1d86066bc449b2b6527f6056048470: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:13.991442 16039 consensus_queue.cc:1048] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a72f9aaff93c4696bd1bdd1b78e0f280" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:13.991694 16054 consensus_queue.cc:1048] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f365650f1de74e8ca759210dba0d2865" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:14.006244 16056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:14.003529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:36819 (local address 127.13.88.131:42773)
0128 15:02:14.003659 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:14.003664 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:14.003804 (+ 140us) server_negotiation.cc:408] Connection header received
0128 15:02:14.003899 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:14.003902 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:14.003943 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:14.004006 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:02:14.004408 (+ 402us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:14.004934 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:14.005621 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:14.005811 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:14.005859 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:02:14.005931 (+ 72us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:02:14.006003 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:14.006098 (+ 95us) server_negotiation.cc:300] Negotiation successful
0128 15:02:14.006144 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260128 15:02:14.006269 16064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:14.003476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:42773 (local address 127.13.88.130:36819)
0128 15:02:14.003724 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:14.003736 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:14.003819 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:14.004028 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:14.004031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:14.004039 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 15:02:14.004297 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:14.004306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:14.005055 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:14.005059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:14.005516 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:14.005526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:14.005651 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:14.005684 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 15:02:14.006014 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 15:02:14.006018 (+ 4us) client_negotiation.cc:769] Sending connection context
0128 15:02:14.006068 (+ 50us) client_negotiation.cc:240] Negotiation successful
0128 15:02:14.006139 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":109,"threads_started":1}
W20260128 15:02:14.006878 16007 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20260128 15:02:15.559825 15428 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:02:15.559963 15428 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:02:15.562019 15428 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 9bb8020800534568b96875932aed51fb: Sending DeleteTablet for 3 replicas of tablet cc8ce026376c47bfbffc0c33f0d3770a
I20260128 15:02:15.562508 15569 tablet_service.cc:1552] Processing DeleteTablet for tablet cc8ce026376c47bfbffc0c33f0d3770a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:15 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:42412
I20260128 15:02:15.562503 15710 tablet_service.cc:1552] Processing DeleteTablet for tablet cc8ce026376c47bfbffc0c33f0d3770a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:15 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:41870
I20260128 15:02:15.562558 15853 tablet_service.cc:1552] Processing DeleteTablet for tablet cc8ce026376c47bfbffc0c33f0d3770a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:15 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:44792
I20260128 15:02:15.562868 16089 tablet_replica.cc:333] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470: stopping tablet replica
I20260128 15:02:15.562891 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 15504
I20260128 15:02:15.563524 16089 raft_consensus.cc:2243] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [term 1 LEADER]: Raft consensus shutting down.
I20260128 15:02:15.563838 16089 raft_consensus.cc:2272] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:15.564373 16089 ts_tablet_manager.cc:1916] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:15.566337 16089 ts_tablet_manager.cc:1929] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260128 15:02:15.566397 16089 log.cc:1199] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-2/wal/wals/cc8ce026376c47bfbffc0c33f0d3770a
I20260128 15:02:15.566619 16089 ts_tablet_manager.cc:1950] T cc8ce026376c47bfbffc0c33f0d3770a P fd1d86066bc449b2b6527f6056048470: Deleting consensus metadata
I20260128 15:02:15.567171 16088 tablet_replica.cc:333] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280: stopping tablet replica
I20260128 15:02:15.567209 15415 catalog_manager.cc:4985] TS fd1d86066bc449b2b6527f6056048470 (127.13.88.131:42773): tablet cc8ce026376c47bfbffc0c33f0d3770a (table test-table [id=b3009b16213c4c67b42c93c9a2f5c17c]) successfully deleted
I20260128 15:02:15.567291 16088 raft_consensus.cc:2243] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:02:15.567396 16088 raft_consensus.cc:2272] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:15.567888 16088 ts_tablet_manager.cc:1916] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:15.569761 16088 ts_tablet_manager.cc:1929] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260128 15:02:15.569873 16088 log.cc:1199] T cc8ce026376c47bfbffc0c33f0d3770a P a72f9aaff93c4696bd1bdd1b78e0f280: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1769612519049359-13666-0/minicluster-data/ts-1/wal/wals/cc8ce026376c47bfbffc0c33f0d3770a
W20260128 15:02:15.569891 15416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.88.129:35019 (error 108)
I20260128 15:02:15.570067 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 15646
I20260128 15:02:15.571370 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:15.570806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:35019 (local address 127.0.0.1:43458)
0128 15:02:15.571233 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:15.571284 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.129:35019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":72,"threads_started":1}
W20260128 15:02:15.571462 15416 catalog_manager.cc:4712] TS f365650f1de74e8ca759210dba0d2865 (127.13.88.129:35019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc8ce026376c47bfbffc0c33f0d3770a: Network error: Client connection negotiation failed: client connection to 127.13.88.129:35019: connect: Connection refused (error 111)
I20260128 15:02:15.575935 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 15788
I20260128 15:02:15.576869 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:15.576022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:40711 (local address 127.0.0.1:57958)
0128 15:02:15.576763 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:15.576805 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.130:40711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":671}
I20260128 15:02:15.581475 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 15398
2026-01-28T15:02:15Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7081 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:02:15 dist-test-slave-m28t krb5kdc[16094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:02:15 dist-test-slave-m28t krb5kdc[16094](info): set up 2 sockets
Jan 28 15:02:15 dist-test-slave-m28t krb5kdc[16094](info): commencing operation
krb5kdc: starting...
W20260128 15:02:17.404340 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.802s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Jan 28 15:02:17 dist-test-slave-m28t krb5kdc[16094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:02:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:17.536056 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:36631
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:36631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:17.639458 16110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:17.639679 16110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:17.639741 16110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:17.643023 16110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:17.643085 16110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:17.643107 16110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:17.643126 16110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:17.643144 16110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:02:17.647627 16110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:36631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:36631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:17.648753 16110 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:17.649636 16110 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:17.655288 16116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:17.655288 16115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:17.655316 16118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:17.655611 16110 server_base.cc:1060] running on GCE node
I20260128 15:02:17.656138 16110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:17.657136 16110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:17.658273 16110 hybrid_clock.cc:648] HybridClock initialized: now 1769612537658257 us; error 37 us; skew 500 ppm
Jan 28 15:02:17 dist-test-slave-m28t krb5kdc[16094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612537, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:17.660693 16110 init.cc:377] Logged in from keytab as kudu/127.13.88.190@KRBTEST.COM (short username kudu)
I20260128 15:02:17.661722 16110 webserver.cc:492] Webserver started at http://127.13.88.190:35841/ using document root <none> and password file <none>
I20260128 15:02:17.662242 16110 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:17.662288 16110 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:17.662472 16110 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:17.664108 16110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "bda630106a124c388275314e0c5f16db"
format_stamp: "Formatted at 2026-01-28 15:02:17 on dist-test-slave-m28t"
server_key: "d8b5a227655d7c377df3785ad2c300d9"
server_key_iv: "1f0f65077c0497c98e98178c5bba3878"
server_key_version: "encryptionkey@0"
I20260128 15:02:17.664572 16110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "bda630106a124c388275314e0c5f16db"
format_stamp: "Formatted at 2026-01-28 15:02:17 on dist-test-slave-m28t"
server_key: "d8b5a227655d7c377df3785ad2c300d9"
server_key_iv: "1f0f65077c0497c98e98178c5bba3878"
server_key_version: "encryptionkey@0"
I20260128 15:02:17.667733 16110 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260128 15:02:17.669786 16125 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:17.670723 16110 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:02:17.670830 16110 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "bda630106a124c388275314e0c5f16db"
format_stamp: "Formatted at 2026-01-28 15:02:17 on dist-test-slave-m28t"
server_key: "d8b5a227655d7c377df3785ad2c300d9"
server_key_iv: "1f0f65077c0497c98e98178c5bba3878"
server_key_version: "encryptionkey@0"
I20260128 15:02:17.670912 16110 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:17.686455 16110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:17.689193 16110 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:17.689368 16110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:17.696305 16177 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:36631 every 8 connection(s)
I20260128 15:02:17.696316 16110 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:36631
I20260128 15:02:17.697459 16110 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:17.700110 16178 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:17.701107 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 16110
I20260128 15:02:17.701233 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:17.701481 13666 external_mini_cluster.cc:1468] Setting key f29f880d4f77561d57d95270f8e92af3
I20260128 15:02:17.706066 16178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db: Bootstrap starting.
I20260128 15:02:17.708570 16178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db: Neither blocks nor log segments found. Creating new log.
Jan 28 15:02:17 dist-test-slave-m28t krb5kdc[16094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:17.709244 16178 log.cc:826] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:17.710932 16178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db: No bootstrap required, opened a new log
I20260128 15:02:17.713048 16181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:17.702622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.13.88.190:36631)
0128 15:02:17.702972 (+ 350us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:17.702982 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:17.703015 (+ 33us) server_negotiation.cc:408] Connection header received
0128 15:02:17.703535 (+ 520us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:17.703553 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:17.703841 (+ 288us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:17.704089 (+ 248us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:17.705099 (+ 1010us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:17.705828 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:17.706610 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:17.706846 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:17.709124 (+ 2278us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:17.709141 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:17.709151 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:17.709185 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:17.710713 (+ 1528us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:17.711109 (+ 396us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:17.711115 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:17.711120 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:17.711194 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:17.711397 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:17.711401 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:17.711403 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:17.711752 (+ 349us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:17.711926 (+ 174us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:17.712200 (+ 274us) server_negotiation.cc:300] Negotiation successful
0128 15:02:17.712467 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":97,"threads_started":1}
I20260128 15:02:17.713461 16178 raft_consensus.cc:359] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda630106a124c388275314e0c5f16db" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 36631 } }
I20260128 15:02:17.713647 16178 raft_consensus.cc:385] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:17.713725 16178 raft_consensus.cc:740] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda630106a124c388275314e0c5f16db, State: Initialized, Role: FOLLOWER
I20260128 15:02:17.714150 16178 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda630106a124c388275314e0c5f16db" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 36631 } }
I20260128 15:02:17.714300 16178 raft_consensus.cc:399] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:17.714359 16178 raft_consensus.cc:493] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:17.714440 16178 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:17.715624 16178 raft_consensus.cc:515] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda630106a124c388275314e0c5f16db" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 36631 } }
I20260128 15:02:17.715915 16178 leader_election.cc:304] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda630106a124c388275314e0c5f16db; no voters:
I20260128 15:02:17.716154 16178 leader_election.cc:290] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:17.716241 16183 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:17.716470 16183 raft_consensus.cc:697] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [term 1 LEADER]: Becoming Leader. State: Replica: bda630106a124c388275314e0c5f16db, State: Running, Role: LEADER
I20260128 15:02:17.716732 16183 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda630106a124c388275314e0c5f16db" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 36631 } }
I20260128 15:02:17.717151 16178 sys_catalog.cc:565] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:17.718075 16185 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda630106a124c388275314e0c5f16db. Latest consensus state: current_term: 1 leader_uuid: "bda630106a124c388275314e0c5f16db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda630106a124c388275314e0c5f16db" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 36631 } } }
I20260128 15:02:17.718180 16185 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:17.718551 16192 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:17.719271 16184 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bda630106a124c388275314e0c5f16db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda630106a124c388275314e0c5f16db" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 36631 } } }
I20260128 15:02:17.719362 16184 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:17.721100 16192 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:17.725981 16192 catalog_manager.cc:1357] Generated new cluster ID: 8c73a123378a4bb78f0332bb7688e5d8
I20260128 15:02:17.726037 16192 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:17.751736 16192 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:17.752704 16192 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:17.762477 16192 catalog_manager.cc:6027] T 00000000000000000000000000000000 P bda630106a124c388275314e0c5f16db: Generated new TSK 0
I20260128 15:02:17.763175 16192 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:17.808917 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:36631
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:17.912189 16206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:17.912395 16206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:17.912487 16206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:17.915697 16206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:17.915759 16206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:17.915851 16206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:02:17.919961 16206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:36631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:17.920984 16206 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:17.921787 16206 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:17.928205 16214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:17.928236 16211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:17.928210 16212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:17.928504 16206 server_base.cc:1060] running on GCE node
I20260128 15:02:17.928861 16206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:17.929374 16206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:17.930531 16206 hybrid_clock.cc:648] HybridClock initialized: now 1769612537930519 us; error 24 us; skew 500 ppm
Jan 28 15:02:17 dist-test-slave-m28t krb5kdc[16094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612537, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:17.932963 16206 init.cc:377] Logged in from keytab as kudu/127.13.88.129@KRBTEST.COM (short username kudu)
I20260128 15:02:17.933960 16206 webserver.cc:492] Webserver started at http://127.13.88.129:42549/ using document root <none> and password file <none>
I20260128 15:02:17.934465 16206 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:17.934514 16206 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:17.934680 16206 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:17.936374 16206 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "49673bc75a144ccb880204c1d53edc21"
format_stamp: "Formatted at 2026-01-28 15:02:17 on dist-test-slave-m28t"
server_key: "601b65ceb95271aa9e3f230c6783a4d8"
server_key_iv: "6bd0885afe6bf8d58fbd685528e1a96a"
server_key_version: "encryptionkey@0"
I20260128 15:02:17.936856 16206 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "49673bc75a144ccb880204c1d53edc21"
format_stamp: "Formatted at 2026-01-28 15:02:17 on dist-test-slave-m28t"
server_key: "601b65ceb95271aa9e3f230c6783a4d8"
server_key_iv: "6bd0885afe6bf8d58fbd685528e1a96a"
server_key_version: "encryptionkey@0"
I20260128 15:02:17.939941 16206 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260128 15:02:17.942085 16221 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:17.943006 16206 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260128 15:02:17.943178 16206 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "49673bc75a144ccb880204c1d53edc21"
format_stamp: "Formatted at 2026-01-28 15:02:17 on dist-test-slave-m28t"
server_key: "601b65ceb95271aa9e3f230c6783a4d8"
server_key_iv: "6bd0885afe6bf8d58fbd685528e1a96a"
server_key_version: "encryptionkey@0"
I20260128 15:02:17.943277 16206 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:17.952494 16206 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:17.954931 16206 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:17.955114 16206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:17.955667 16206 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:17.956619 16206 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:17.956669 16206 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:17.956722 16206 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:17.956754 16206 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:17.965602 16206 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:41733
I20260128 15:02:17.965629 16334 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:41733 every 8 connection(s)
I20260128 15:02:17.966589 16206 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:02:17.973861 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 16206
I20260128 15:02:17.973973 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:02:17.974198 13666 external_mini_cluster.cc:1468] Setting key 4a314fe493785b80b41509264da98ef2
Jan 28 15:02:17 dist-test-slave-m28t krb5kdc[16094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612537, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:17.979125 16181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:17.968355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:58023 (local address 127.13.88.190:36631)
0128 15:02:17.968507 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:17.968511 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:17.969246 (+ 735us) server_negotiation.cc:408] Connection header received
0128 15:02:17.970094 (+ 848us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:17.970098 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:17.970158 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:17.970228 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:17.971900 (+ 1672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:17.972409 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:17.973091 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:17.973254 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:17.975559 (+ 2305us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:17.975576 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:17.975579 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:17.975601 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:17.977024 (+ 1423us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:17.977574 (+ 550us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:17.977577 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:17.977578 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:17.977625 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:17.977961 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:17.977964 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:17.977965 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:17.978140 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:17.978236 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:17.978860 (+ 624us) server_negotiation.cc:300] Negotiation successful
0128 15:02:17.978988 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260128 15:02:17.979982 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:17.968676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:36631 (local address 127.13.88.129:58023)
0128 15:02:17.969111 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:17.969148 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:17.969897 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:17.970392 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:17.970401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:17.970798 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:17.971742 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:17.971758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:17.972559 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:17.972562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:17.972959 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:17.972968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:17.973178 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:17.973719 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:17.973745 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:17.975383 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:17.977145 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:17.977151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:17.977161 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:17.977472 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:17.977703 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:17.977706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:17.977709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:17.977840 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:17.978250 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:17.978257 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:17.978566 (+ 309us) client_negotiation.cc:769] Sending connection context
0128 15:02:17.978808 (+ 242us) client_negotiation.cc:240] Negotiation successful
0128 15:02:17.979102 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
I20260128 15:02:17.981225 16335 heartbeater.cc:344] Connected to a master server at 127.13.88.190:36631
I20260128 15:02:17.981504 16335 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:17.982172 16335 heartbeater.cc:507] Master 127.13.88.190:36631 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.88.130@KRBTEST.COM; defaulting to no policy
I20260128 15:02:17.983737 16142 ts_manager.cc:194] Registered new tserver with Master: 49673bc75a144ccb880204c1d53edc21 (127.13.88.129:41733)
I20260128 15:02:17.985047 16142 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.129@KRBTEST.COM'} at 127.13.88.129:58023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:18.019366 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:36631
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:18.121559 16342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:18.121758 16342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:18.121819 16342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:18.124980 16342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:18.125033 16342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:18.125106 16342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:02:18.129139 16342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:36631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:18.130097 16342 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:18.130869 16342 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:18.136991 16347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:18.137004 16348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:18.136999 16350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:18.137252 16342 server_base.cc:1060] running on GCE node
I20260128 15:02:18.137681 16342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:18.138167 16342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:18.139328 16342 hybrid_clock.cc:648] HybridClock initialized: now 1769612538139314 us; error 28 us; skew 500 ppm
Jan 28 15:02:18 dist-test-slave-m28t krb5kdc[16094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612538, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:18.141783 16342 init.cc:377] Logged in from keytab as kudu/127.13.88.130@KRBTEST.COM (short username kudu)
I20260128 15:02:18.142769 16342 webserver.cc:492] Webserver started at http://127.13.88.130:38533/ using document root <none> and password file <none>
I20260128 15:02:18.143280 16342 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:18.143328 16342 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:18.143491 16342 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:18.145220 16342 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "7bd435f7c62b4ead9e2762c4c1217fcd"
format_stamp: "Formatted at 2026-01-28 15:02:18 on dist-test-slave-m28t"
server_key: "a47904900107db90d793e0534f0bfab5"
server_key_iv: "daaa624bdb20d2b37a2fb66fbcf19f0d"
server_key_version: "encryptionkey@0"
I20260128 15:02:18.145663 16342 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "7bd435f7c62b4ead9e2762c4c1217fcd"
format_stamp: "Formatted at 2026-01-28 15:02:18 on dist-test-slave-m28t"
server_key: "a47904900107db90d793e0534f0bfab5"
server_key_iv: "daaa624bdb20d2b37a2fb66fbcf19f0d"
server_key_version: "encryptionkey@0"
I20260128 15:02:18.149011 16342 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260128 15:02:18.151254 16357 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:18.152272 16342 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260128 15:02:18.152387 16342 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "7bd435f7c62b4ead9e2762c4c1217fcd"
format_stamp: "Formatted at 2026-01-28 15:02:18 on dist-test-slave-m28t"
server_key: "a47904900107db90d793e0534f0bfab5"
server_key_iv: "daaa624bdb20d2b37a2fb66fbcf19f0d"
server_key_version: "encryptionkey@0"
I20260128 15:02:18.152521 16342 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:18.167804 16342 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:18.170372 16342 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:18.170545 16342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:18.171074 16342 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:18.171940 16342 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:18.171989 16342 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:18.172039 16342 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:18.172067 16342 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:18.180395 16342 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:38935
I20260128 15:02:18.180438 16470 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:38935 every 8 connection(s)
I20260128 15:02:18.181329 16342 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
I20260128 15:02:18.184504 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 16342
I20260128 15:02:18.184619 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:02:18.184895 13666 external_mini_cluster.cc:1468] Setting key 8e532eba2b2df1bafdb9ca796521d09f
Jan 28 15:02:18 dist-test-slave-m28t krb5kdc[16094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612538, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.88.131@KRBTEST.COM; defaulting to no policy
I20260128 15:02:18.193869 16181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:18.182934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:58117 (local address 127.13.88.190:36631)
0128 15:02:18.183085 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:18.183090 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:18.183801 (+ 711us) server_negotiation.cc:408] Connection header received
0128 15:02:18.184742 (+ 941us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:18.184745 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:18.184796 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:18.184865 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:18.186786 (+ 1921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:18.187271 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:18.188011 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:18.188197 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:18.190316 (+ 2119us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:18.190335 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:18.190339 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:18.190371 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:18.192068 (+ 1697us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:18.192528 (+ 460us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:18.192532 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:18.192533 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:18.192591 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:18.192853 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:18.192856 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:18.192859 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:18.193051 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:18.193152 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:18.193620 (+ 468us) server_negotiation.cc:300] Negotiation successful
0128 15:02:18.193749 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260128 15:02:18.194463 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:18.183209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:36631 (local address 127.13.88.130:58117)
0128 15:02:18.183648 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:18.183686 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:18.184530 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:18.185048 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:18.185058 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:18.185542 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:18.186648 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:18.186667 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:18.187374 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:18.187378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:18.187899 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:18.187907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:18.188110 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:18.188756 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:18.188779 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:18.190174 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:18.192177 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:18.192182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:18.192191 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:18.192406 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:18.192677 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:18.192679 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:18.192681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:18.192763 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:18.193136 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:18.193141 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:18.193357 (+ 216us) client_negotiation.cc:769] Sending connection context
0128 15:02:18.193536 (+ 179us) client_negotiation.cc:240] Negotiation successful
0128 15:02:18.193753 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":103,"threads_started":1}
I20260128 15:02:18.195464 16471 heartbeater.cc:344] Connected to a master server at 127.13.88.190:36631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.131@KRBTEST.COM" created.
I20260128 15:02:18.195729 16471 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:18.196306 16471 heartbeater.cc:507] Master 127.13.88.190:36631 requested a full tablet report, sending...
I20260128 15:02:18.197351 16142 ts_manager.cc:194] Registered new tserver with Master: 7bd435f7c62b4ead9e2762c4c1217fcd (127.13.88.130:38935)
I20260128 15:02:18.198000 16142 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.130@KRBTEST.COM'} at 127.13.88.130:58117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:18.228366 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:36631
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:18.332901 16478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:18.333106 16478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:18.333163 16478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:18.336552 16478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:18.336613 16478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:18.336694 16478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:02:18.340857 16478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:36631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:18.341861 16478 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:18.342645 16478 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:18.348941 16484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:18.349012 16486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:18.349016 16478 server_base.cc:1060] running on GCE node
W20260128 15:02:18.348948 16483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:18.349431 16478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:18.349946 16478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:18.351089 16478 hybrid_clock.cc:648] HybridClock initialized: now 1769612538351077 us; error 26 us; skew 500 ppm
Jan 28 15:02:18 dist-test-slave-m28t krb5kdc[16094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612538, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:18.353477 16478 init.cc:377] Logged in from keytab as kudu/127.13.88.131@KRBTEST.COM (short username kudu)
I20260128 15:02:18.354468 16478 webserver.cc:492] Webserver started at http://127.13.88.131:42419/ using document root <none> and password file <none>
I20260128 15:02:18.354987 16478 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:18.355039 16478 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:18.355203 16478 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:18.356918 16478 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "7fbacc77481a438c8507ccfa3d878ffb"
format_stamp: "Formatted at 2026-01-28 15:02:18 on dist-test-slave-m28t"
server_key: "f9cf26e9e60aabe5fb4a6fe5f8d86998"
server_key_iv: "42b0226ebf08bdd39ec012b106ddb520"
server_key_version: "encryptionkey@0"
I20260128 15:02:18.357343 16478 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "7fbacc77481a438c8507ccfa3d878ffb"
format_stamp: "Formatted at 2026-01-28 15:02:18 on dist-test-slave-m28t"
server_key: "f9cf26e9e60aabe5fb4a6fe5f8d86998"
server_key_iv: "42b0226ebf08bdd39ec012b106ddb520"
server_key_version: "encryptionkey@0"
I20260128 15:02:18.360411 16478 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260128 15:02:18.362442 16493 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:18.363430 16478 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:18.363523 16478 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "7fbacc77481a438c8507ccfa3d878ffb"
format_stamp: "Formatted at 2026-01-28 15:02:18 on dist-test-slave-m28t"
server_key: "f9cf26e9e60aabe5fb4a6fe5f8d86998"
server_key_iv: "42b0226ebf08bdd39ec012b106ddb520"
server_key_version: "encryptionkey@0"
I20260128 15:02:18.363612 16478 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:18.375658 16478 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:18.377990 16478 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:18.378146 16478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:18.378643 16478 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:18.379444 16478 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:18.379491 16478 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:18.379547 16478 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:18.379576 16478 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:18.388185 16478 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:37457
I20260128 15:02:18.388216 16606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:37457 every 8 connection(s)
I20260128 15:02:18.389150 16478 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
I20260128 15:02:18.393666 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 16478
I20260128 15:02:18.393754 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:02:18.393963 13666 external_mini_cluster.cc:1468] Setting key d3e50cc3cc2081cfd16045cfd2f243b2
Jan 28 15:02:18 dist-test-slave-m28t krb5kdc[16094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612538, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:18.400204 16181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:18.390821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:57241 (local address 127.13.88.190:36631)
0128 15:02:18.390989 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:18.390995 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:18.391641 (+ 646us) server_negotiation.cc:408] Connection header received
0128 15:02:18.392408 (+ 767us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:18.392411 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:18.392495 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:18.392570 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:18.394170 (+ 1600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:18.394738 (+ 568us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:18.395416 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:18.395622 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:18.397662 (+ 2040us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:18.397677 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:18.397679 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:18.397706 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:18.398769 (+ 1063us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:18.399163 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:18.399166 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:18.399167 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:18.399204 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:18.399389 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:18.399390 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:18.399392 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:18.399522 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:18.399596 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:18.399996 (+ 400us) server_negotiation.cc:300] Negotiation successful
0128 15:02:18.400088 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260128 15:02:18.400832 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:18.391089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:36631 (local address 127.13.88.131:57241)
0128 15:02:18.391495 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:18.391530 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:18.392238 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:18.392714 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:18.392722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:18.393081 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:18.394027 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:18.394045 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:18.394854 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:18.394857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:18.395309 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:18.395318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:18.395529 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:18.396100 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:18.396124 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:18.397526 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:18.398871 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:18.398876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:18.398884 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:18.399094 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:18.399258 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:18.399261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:18.399262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:18.399333 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:18.399588 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:18.399592 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:18.399808 (+ 216us) client_negotiation.cc:769] Sending connection context
0128 15:02:18.399967 (+ 159us) client_negotiation.cc:240] Negotiation successful
0128 15:02:18.400166 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":100,"threads_started":1}
I20260128 15:02:18.401746 16607 heartbeater.cc:344] Connected to a master server at 127.13.88.190:36631
I20260128 15:02:18.401973 16607 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:18.402443 16607 heartbeater.cc:507] Master 127.13.88.190:36631 requested a full tablet report, sending...
I20260128 15:02:18.403352 16142 ts_manager.cc:194] Registered new tserver with Master: 7fbacc77481a438c8507ccfa3d878ffb (127.13.88.131:37457)
I20260128 15:02:18.403875 16142 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.131@KRBTEST.COM'} at 127.13.88.131:57241
I20260128 15:02:18.407209 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260128 15:02:18.418085 16181 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:02:18.416709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56154 (local address 127.13.88.190:36631)
0128 15:02:18.416886 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:18.416891 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:18.417010 (+ 119us) server_negotiation.cc:408] Connection header received
0128 15:02:18.417140 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:18.417143 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:18.417188 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:18.417254 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:18.417932 (+ 678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56154: BlockingRecv error: recv got EOF from 127.0.0.1:56154 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260128 15:02:18.419153 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 16206
I20260128 15:02:18.424079 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 16342
I20260128 15:02:18.428721 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 16478
I20260128 15:02:18.433205 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 16110
2026-01-28T15:02:18Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2850 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:02:18 dist-test-slave-m28t krb5kdc[16619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:02:18 dist-test-slave-m28t krb5kdc[16619](info): set up 2 sockets
Jan 28 15:02:18 dist-test-slave-m28t krb5kdc[16619](info): commencing operation
krb5kdc: starting...
W20260128 15:02:20.256497 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.803s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Jan 28 15:02:20 dist-test-slave-m28t krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:02:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:20.380543 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:33097
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:33097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:20.480947 16635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:20.481156 16635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:20.481225 16635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:20.484345 16635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:20.484401 16635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:20.484450 16635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:20.484485 16635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:20.484512 16635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:02:20.488525 16635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:33097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:33097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:20.489463 16635 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:20.490228 16635 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:20.495319 16640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:20.495370 16643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:20.495323 16641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:20.495592 16635 server_base.cc:1060] running on GCE node
I20260128 15:02:20.496095 16635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:20.496963 16635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:20.498104 16635 hybrid_clock.cc:648] HybridClock initialized: now 1769612540498094 us; error 29 us; skew 500 ppm
Jan 28 15:02:20 dist-test-slave-m28t krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612540, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:20.500320 16635 init.cc:377] Logged in from keytab as kudu/127.13.88.190@KRBTEST.COM (short username kudu)
I20260128 15:02:20.501307 16635 webserver.cc:492] Webserver started at http://127.13.88.190:35945/ using document root <none> and password file <none>
I20260128 15:02:20.501797 16635 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:20.501848 16635 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:20.502012 16635 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:20.503584 16635 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "a12fd11d1afe4a4bb849f151a0089e62"
format_stamp: "Formatted at 2026-01-28 15:02:20 on dist-test-slave-m28t"
server_key: "c24e60513d9bf38671d3ee3be1ce65d6"
server_key_iv: "de87ba18ac7fc96d54e56d2d58392bab"
server_key_version: "encryptionkey@0"
I20260128 15:02:20.504019 16635 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "a12fd11d1afe4a4bb849f151a0089e62"
format_stamp: "Formatted at 2026-01-28 15:02:20 on dist-test-slave-m28t"
server_key: "c24e60513d9bf38671d3ee3be1ce65d6"
server_key_iv: "de87ba18ac7fc96d54e56d2d58392bab"
server_key_version: "encryptionkey@0"
I20260128 15:02:20.507128 16635 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260128 15:02:20.509186 16650 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:20.510079 16635 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:20.510178 16635 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "a12fd11d1afe4a4bb849f151a0089e62"
format_stamp: "Formatted at 2026-01-28 15:02:20 on dist-test-slave-m28t"
server_key: "c24e60513d9bf38671d3ee3be1ce65d6"
server_key_iv: "de87ba18ac7fc96d54e56d2d58392bab"
server_key_version: "encryptionkey@0"
I20260128 15:02:20.510254 16635 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:20.529809 16635 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:20.532266 16635 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:20.532452 16635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:20.539059 16635 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:33097
I20260128 15:02:20.539074 16702 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:33097 every 8 connection(s)
I20260128 15:02:20.540024 16635 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:20.542598 16703 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:20.544992 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 16635
I20260128 15:02:20.545094 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:20.545292 13666 external_mini_cluster.cc:1468] Setting key e8644a7b17b1d9ac5bf9c411cbe44ffc
I20260128 15:02:20.548149 16703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62: Bootstrap starting.
I20260128 15:02:20.550473 16703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:20.551270 16703 log.cc:826] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62: Log is configured to *not* fsync() on all Append() calls
Jan 28 15:02:20 dist-test-slave-m28t krb5kdc[16619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:20.553188 16703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62: No bootstrap required, opened a new log
I20260128 15:02:20.555691 16703 raft_consensus.cc:359] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12fd11d1afe4a4bb849f151a0089e62" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 33097 } }
I20260128 15:02:20.555912 16703 raft_consensus.cc:385] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:20.555974 16703 raft_consensus.cc:740] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12fd11d1afe4a4bb849f151a0089e62, State: Initialized, Role: FOLLOWER
I20260128 15:02:20.556411 16703 consensus_queue.cc:260] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12fd11d1afe4a4bb849f151a0089e62" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 33097 } }
I20260128 15:02:20.556581 16703 raft_consensus.cc:399] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:20.556658 16703 raft_consensus.cc:493] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:20.556766 16703 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:20.557545 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:20.546357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48130 (local address 127.13.88.190:33097)
0128 15:02:20.546707 (+ 350us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:20.546715 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:20.546749 (+ 34us) server_negotiation.cc:408] Connection header received
0128 15:02:20.547252 (+ 503us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:20.547268 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:20.547531 (+ 263us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:20.547787 (+ 256us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:20.549397 (+ 1610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:20.550125 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:20.550962 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:20.551162 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:20.553386 (+ 2224us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:20.553403 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:20.553412 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:20.553443 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:20.554996 (+ 1553us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:20.555428 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:20.555433 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:20.555437 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:20.555514 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:20.555744 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:20.555746 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:20.555748 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:20.556131 (+ 383us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:20.556340 (+ 209us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:20.556652 (+ 312us) server_negotiation.cc:300] Negotiation successful
0128 15:02:20.556927 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":96,"threads_started":1}
I20260128 15:02:20.557638 16703 raft_consensus.cc:515] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12fd11d1afe4a4bb849f151a0089e62" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 33097 } }
I20260128 15:02:20.558013 16703 leader_election.cc:304] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a12fd11d1afe4a4bb849f151a0089e62; no voters:
I20260128 15:02:20.558316 16703 leader_election.cc:290] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:20.558406 16708 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:20.559499 16703 sys_catalog.cc:565] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:20.559617 16708 raft_consensus.cc:697] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [term 1 LEADER]: Becoming Leader. State: Replica: a12fd11d1afe4a4bb849f151a0089e62, State: Running, Role: LEADER
I20260128 15:02:20.559922 16708 consensus_queue.cc:237] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12fd11d1afe4a4bb849f151a0089e62" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 33097 } }
I20260128 15:02:20.561251 16710 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a12fd11d1afe4a4bb849f151a0089e62. Latest consensus state: current_term: 1 leader_uuid: "a12fd11d1afe4a4bb849f151a0089e62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12fd11d1afe4a4bb849f151a0089e62" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 33097 } } }
I20260128 15:02:20.561450 16710 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:20.561538 16709 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a12fd11d1afe4a4bb849f151a0089e62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12fd11d1afe4a4bb849f151a0089e62" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 33097 } } }
I20260128 15:02:20.561646 16709 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62 [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:20.561921 16720 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:20.565284 16720 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:20.570639 16720 catalog_manager.cc:1357] Generated new cluster ID: 7a89682d231d420eb23cba9550e84f9a
I20260128 15:02:20.570792 16720 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:20.589854 16720 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:20.590788 16720 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:20.597201 16720 catalog_manager.cc:6027] T 00000000000000000000000000000000 P a12fd11d1afe4a4bb849f151a0089e62: Generated new TSK 0
I20260128 15:02:20.597886 16720 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:20.647248 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:33097
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:20.749861 16731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:20.750083 16731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:20.750156 16731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:20.753392 16731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:20.753454 16731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:20.753558 16731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:02:20.757611 16731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:33097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:20.758566 16731 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:20.759348 16731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:20.765563 16737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:20.765645 16731 server_base.cc:1060] running on GCE node
W20260128 15:02:20.765519 16739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:20.765561 16736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:20.766145 16731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:20.766642 16731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:20.767803 16731 hybrid_clock.cc:648] HybridClock initialized: now 1769612540767791 us; error 26 us; skew 500 ppm
Jan 28 15:02:20 dist-test-slave-m28t krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612540, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:20.770227 16731 init.cc:377] Logged in from keytab as kudu/127.13.88.129@KRBTEST.COM (short username kudu)
I20260128 15:02:20.771206 16731 webserver.cc:492] Webserver started at http://127.13.88.129:33887/ using document root <none> and password file <none>
I20260128 15:02:20.771806 16731 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:20.771857 16731 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:20.772022 16731 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:20.773699 16731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "2e26f5f3d86b44efbcb74708914a3b1f"
format_stamp: "Formatted at 2026-01-28 15:02:20 on dist-test-slave-m28t"
server_key: "c286e7bfc28faf033276e3dd12129949"
server_key_iv: "603d2ba2e3341c222e9b4f4dae7b13d6"
server_key_version: "encryptionkey@0"
I20260128 15:02:20.774140 16731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "2e26f5f3d86b44efbcb74708914a3b1f"
format_stamp: "Formatted at 2026-01-28 15:02:20 on dist-test-slave-m28t"
server_key: "c286e7bfc28faf033276e3dd12129949"
server_key_iv: "603d2ba2e3341c222e9b4f4dae7b13d6"
server_key_version: "encryptionkey@0"
I20260128 15:02:20.777302 16731 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260128 15:02:20.779347 16746 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:20.780296 16731 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:20.780397 16731 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "2e26f5f3d86b44efbcb74708914a3b1f"
format_stamp: "Formatted at 2026-01-28 15:02:20 on dist-test-slave-m28t"
server_key: "c286e7bfc28faf033276e3dd12129949"
server_key_iv: "603d2ba2e3341c222e9b4f4dae7b13d6"
server_key_version: "encryptionkey@0"
I20260128 15:02:20.780510 16731 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:20.793737 16731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:20.796231 16731 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:20.796387 16731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:20.796913 16731 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:20.797734 16731 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:20.797784 16731 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:20.797822 16731 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:20.797842 16731 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:20.806124 16731 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:46169
I20260128 15:02:20.806164 16859 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:46169 every 8 connection(s)
I20260128 15:02:20.807039 16731 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:02:20.812222 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 16731
I20260128 15:02:20.812310 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:02:20.812572 13666 external_mini_cluster.cc:1468] Setting key e8accd95e8a58529185cc9f73838b363
Jan 28 15:02:20 dist-test-slave-m28t krb5kdc[16619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612540, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:20.819180 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:20.808707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:48761 (local address 127.13.88.190:33097)
0128 15:02:20.808865 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:20.808869 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:20.809476 (+ 607us) server_negotiation.cc:408] Connection header received
0128 15:02:20.810270 (+ 794us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:20.810274 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:20.810329 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:20.810400 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:20.811999 (+ 1599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:20.812592 (+ 593us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:20.813255 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:20.813404 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:20.815801 (+ 2397us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:20.815821 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:20.815823 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:20.815847 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:20.817257 (+ 1410us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:20.817703 (+ 446us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:20.817707 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:20.817709 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:20.817758 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:20.818075 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:20.818079 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:20.818081 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:20.818281 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:20.818385 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:20.818879 (+ 494us) server_negotiation.cc:300] Negotiation successful
0128 15:02:20.819009 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260128 15:02:20.819757 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:20.808967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:33097 (local address 127.13.88.129:48761)
0128 15:02:20.809345 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:20.809378 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:20.810081 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:20.810563 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:20.810571 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:20.810949 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:20.811857 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:20.811871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:20.812696 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:20.812700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:20.813149 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:20.813157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:20.813355 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:20.814016 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:20.814045 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:20.815625 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:20.817367 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:20.817372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:20.817381 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:20.817608 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:20.817879 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:20.817881 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:20.817883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:20.817981 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:20.818399 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:20.818404 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:20.818636 (+ 232us) client_negotiation.cc:769] Sending connection context
0128 15:02:20.818822 (+ 186us) client_negotiation.cc:240] Negotiation successful
0128 15:02:20.819033 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":99,"threads_started":1}
I20260128 15:02:20.820967 16860 heartbeater.cc:344] Connected to a master server at 127.13.88.190:33097
WARNING: no policy specified for kudu/127.13.88.130@KRBTEST.COM; defaulting to no policy
I20260128 15:02:20.821242 16860 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:20.821765 16860 heartbeater.cc:507] Master 127.13.88.190:33097 requested a full tablet report, sending...
I20260128 15:02:20.823199 16667 ts_manager.cc:194] Registered new tserver with Master: 2e26f5f3d86b44efbcb74708914a3b1f (127.13.88.129:46169)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.130@KRBTEST.COM" created.
I20260128 15:02:20.824627 16667 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.129@KRBTEST.COM'} at 127.13.88.129:48761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:20.858549 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:33097
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:20.960155 16867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:20.960394 16867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:20.960497 16867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:20.963754 16867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:20.963825 16867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:20.963929 16867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:02:20.968168 16867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:33097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:20.969190 16867 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:20.969990 16867 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:20.976248 16872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:20.976259 16873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:20.976240 16875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:20.976382 16867 server_base.cc:1060] running on GCE node
I20260128 15:02:20.976742 16867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:20.977221 16867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:20.978367 16867 hybrid_clock.cc:648] HybridClock initialized: now 1769612540978348 us; error 31 us; skew 500 ppm
Jan 28 15:02:20 dist-test-slave-m28t krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612540, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:20.980991 16867 init.cc:377] Logged in from keytab as kudu/127.13.88.130@KRBTEST.COM (short username kudu)
I20260128 15:02:20.981979 16867 webserver.cc:492] Webserver started at http://127.13.88.130:42967/ using document root <none> and password file <none>
I20260128 15:02:20.982488 16867 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:20.982545 16867 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:20.982714 16867 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:20.984406 16867 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "36306290790e42b2912d195345bafdfd"
format_stamp: "Formatted at 2026-01-28 15:02:20 on dist-test-slave-m28t"
server_key: "d16efd192e85a4fb586c9f5281a7d095"
server_key_iv: "7a4f721b3af26d6ca7748ce1ce3724d6"
server_key_version: "encryptionkey@0"
I20260128 15:02:20.984885 16867 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "36306290790e42b2912d195345bafdfd"
format_stamp: "Formatted at 2026-01-28 15:02:20 on dist-test-slave-m28t"
server_key: "d16efd192e85a4fb586c9f5281a7d095"
server_key_iv: "7a4f721b3af26d6ca7748ce1ce3724d6"
server_key_version: "encryptionkey@0"
I20260128 15:02:20.988031 16867 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260128 15:02:20.990118 16882 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:20.991063 16867 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260128 15:02:20.991164 16867 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "36306290790e42b2912d195345bafdfd"
format_stamp: "Formatted at 2026-01-28 15:02:20 on dist-test-slave-m28t"
server_key: "d16efd192e85a4fb586c9f5281a7d095"
server_key_iv: "7a4f721b3af26d6ca7748ce1ce3724d6"
server_key_version: "encryptionkey@0"
I20260128 15:02:20.991243 16867 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:21.010489 16867 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:21.013116 16867 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:21.013289 16867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:21.013828 16867 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:21.014739 16867 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:21.014786 16867 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:21.014838 16867 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:21.014863 16867 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:21.023159 16867 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:42717
I20260128 15:02:21.023188 16995 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:42717 every 8 connection(s)
I20260128 15:02:21.024076 16867 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
Jan 28 15:02:21 dist-test-slave-m28t krb5kdc[16619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612540, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:21.033653 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 16867
I20260128 15:02:21.033748 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:02:21.034034 13666 external_mini_cluster.cc:1468] Setting key fb44d73304af8ed17246b578ab8dfabf
I20260128 15:02:21.035800 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:21.025737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:34887 (local address 127.13.88.190:33097)
0128 15:02:21.025883 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:21.025887 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:21.026531 (+ 644us) server_negotiation.cc:408] Connection header received
0128 15:02:21.027343 (+ 812us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:21.027347 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:21.027405 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:21.027478 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:21.029149 (+ 1671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:21.029649 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:21.030256 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:21.030433 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:21.032518 (+ 2085us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:21.032545 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:21.032548 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:21.032575 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:21.033699 (+ 1124us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:21.034270 (+ 571us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:21.034274 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:21.034276 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:21.034329 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:21.034664 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:21.034668 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:21.034669 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:21.034841 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:21.034934 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:21.035493 (+ 559us) server_negotiation.cc:300] Negotiation successful
0128 15:02:21.035646 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260128 15:02:21.036469 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:21.025986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:33097 (local address 127.13.88.130:34887)
0128 15:02:21.026382 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:21.026418 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:21.027142 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:21.027641 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:21.027650 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:21.028038 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:21.029001 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:21.029016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:21.029769 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:21.029772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:21.030148 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:21.030155 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:21.030360 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:21.030901 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:21.030927 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:21.032340 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:21.033823 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:21.033829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:21.033846 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:21.034175 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:21.034444 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:21.034449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:21.034451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:21.034582 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:21.034949 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:21.034957 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:21.035204 (+ 247us) client_negotiation.cc:769] Sending connection context
0128 15:02:21.035429 (+ 225us) client_negotiation.cc:240] Negotiation successful
0128 15:02:21.035680 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":97,"threads_started":1}
I20260128 15:02:21.037657 16996 heartbeater.cc:344] Connected to a master server at 127.13.88.190:33097
I20260128 15:02:21.037894 16996 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:21.038401 16996 heartbeater.cc:507] Master 127.13.88.190:33097 requested a full tablet report, sending...
I20260128 15:02:21.039400 16667 ts_manager.cc:194] Registered new tserver with Master: 36306290790e42b2912d195345bafdfd (127.13.88.130:42717)
I20260128 15:02:21.039953 16667 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.130@KRBTEST.COM'} at 127.13.88.130:34887
WARNING: no policy specified for kudu/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:21.076536 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:33097
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:21.179157 17003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:21.179373 17003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:21.179448 17003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:21.182662 17003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:21.182718 17003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:21.182811 17003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:02:21.186803 17003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:33097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:21.187776 17003 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:21.188596 17003 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:21.194487 17009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:21.194484 17008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:21.194824 17003 server_base.cc:1060] running on GCE node
W20260128 15:02:21.194845 17011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:21.195195 17003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:21.195782 17003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:21.196935 17003 hybrid_clock.cc:648] HybridClock initialized: now 1769612541196924 us; error 24 us; skew 500 ppm
Jan 28 15:02:21 dist-test-slave-m28t krb5kdc[16619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612541, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:21.199357 17003 init.cc:377] Logged in from keytab as kudu/127.13.88.131@KRBTEST.COM (short username kudu)
I20260128 15:02:21.200306 17003 webserver.cc:492] Webserver started at http://127.13.88.131:34279/ using document root <none> and password file <none>
I20260128 15:02:21.200861 17003 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:21.200910 17003 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:21.201071 17003 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:21.202672 17003 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "ae391ea19fcc4f03b3a2196710036e11"
format_stamp: "Formatted at 2026-01-28 15:02:21 on dist-test-slave-m28t"
server_key: "a90368d63e5c054e2597ec6f634ea93f"
server_key_iv: "2db87ba0da2148cdc67efd151c53370d"
server_key_version: "encryptionkey@0"
I20260128 15:02:21.203092 17003 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "ae391ea19fcc4f03b3a2196710036e11"
format_stamp: "Formatted at 2026-01-28 15:02:21 on dist-test-slave-m28t"
server_key: "a90368d63e5c054e2597ec6f634ea93f"
server_key_iv: "2db87ba0da2148cdc67efd151c53370d"
server_key_version: "encryptionkey@0"
I20260128 15:02:21.206271 17003 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260128 15:02:21.208266 17018 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:21.209249 17003 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:21.209342 17003 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "ae391ea19fcc4f03b3a2196710036e11"
format_stamp: "Formatted at 2026-01-28 15:02:21 on dist-test-slave-m28t"
server_key: "a90368d63e5c054e2597ec6f634ea93f"
server_key_iv: "2db87ba0da2148cdc67efd151c53370d"
server_key_version: "encryptionkey@0"
I20260128 15:02:21.209429 17003 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:21.227267 17003 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:21.229533 17003 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:21.229691 17003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:21.230194 17003 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:21.231014 17003 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:21.231056 17003 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:21.231108 17003 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:21.231137 17003 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:21.239444 17003 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:45491
I20260128 15:02:21.239465 17131 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:45491 every 8 connection(s)
I20260128 15:02:21.240322 17003 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
I20260128 15:02:21.241530 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 17003
I20260128 15:02:21.241647 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:02:21.241858 13666 external_mini_cluster.cc:1468] Setting key 832942fc14762f640fbdc64549648315
Jan 28 15:02:21 dist-test-slave-m28t krb5kdc[16619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612541, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:21.251471 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:21.241996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:53915 (local address 127.13.88.190:33097)
0128 15:02:21.242144 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:21.242149 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:21.242826 (+ 677us) server_negotiation.cc:408] Connection header received
0128 15:02:21.243703 (+ 877us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:21.243707 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:21.243763 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:21.243845 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:21.245445 (+ 1600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:21.245941 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:21.246471 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:21.246603 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:21.248689 (+ 2086us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:21.248703 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:21.248706 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:21.248729 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:21.249769 (+ 1040us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:21.250199 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:21.250202 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:21.250203 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:21.250244 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:21.250505 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:21.250508 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:21.250510 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:21.250654 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:21.250736 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:21.251205 (+ 469us) server_negotiation.cc:300] Negotiation successful
0128 15:02:21.251321 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260128 15:02:21.252041 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:21.242268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:33097 (local address 127.13.88.131:53915)
0128 15:02:21.242693 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:21.242739 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:21.243498 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:21.244006 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:21.244014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:21.244380 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:21.245322 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:21.245335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:21.246022 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:21.246025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:21.246394 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:21.246400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:21.246545 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:21.247146 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:21.247168 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:21.248550 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:21.249874 (+ 1324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:21.249880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:21.249890 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:21.250106 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:21.250326 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:21.250329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:21.250331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:21.250422 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:21.250749 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:21.250755 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:21.250981 (+ 226us) client_negotiation.cc:769] Sending connection context
0128 15:02:21.251152 (+ 171us) client_negotiation.cc:240] Negotiation successful
0128 15:02:21.251389 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":123,"threads_started":1}
I20260128 15:02:21.252971 17132 heartbeater.cc:344] Connected to a master server at 127.13.88.190:33097
I20260128 15:02:21.253183 17132 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:21.253638 17132 heartbeater.cc:507] Master 127.13.88.190:33097 requested a full tablet report, sending...
I20260128 15:02:21.254606 16666 ts_manager.cc:194] Registered new tserver with Master: ae391ea19fcc4f03b3a2196710036e11 (127.13.88.131:45491)
I20260128 15:02:21.255290 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260128 15:02:21.255363 16666 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.131@KRBTEST.COM'} at 127.13.88.131:53915
W20260128 15:02:21.380265 16706 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:02:21.377054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.13.88.190:33097)
0128 15:02:21.377186 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:21.377190 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:21.378077 (+ 887us) server_negotiation.cc:408] Connection header received
0128 15:02:21.378908 (+ 831us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:21.378912 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:21.378963 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:21.379025 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:21.380130 (+ 1105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48146: BlockingRecv error: recv got EOF from 127.0.0.1:48146 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260128 15:02:21.381429 16706 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:02:21.380796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.13.88.190:33097)
0128 15:02:21.380902 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:21.380905 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:21.380916 (+ 11us) server_negotiation.cc:408] Connection header received
0128 15:02:21.380989 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:21.380991 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:21.381026 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:21.381089 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:21.381344 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48156: BlockingRecv error: recv got EOF from 127.0.0.1:48156 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260128 15:02:21.382297 16706 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:02:21.381736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48170 (local address 127.13.88.190:33097)
0128 15:02:21.381830 (+ 94us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:21.381833 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:21.381850 (+ 17us) server_negotiation.cc:408] Connection header received
0128 15:02:21.381913 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:21.381915 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:21.381952 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:21.382010 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:21.382235 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48170: BlockingRecv error: recv got EOF from 127.0.0.1:48170 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260128 15:02:21.385778 16706 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:02:21.385132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48186 (local address 127.13.88.190:33097)
0128 15:02:21.385240 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:21.385245 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:21.385257 (+ 12us) server_negotiation.cc:408] Connection header received
0128 15:02:21.385289 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:21.385291 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:21.385322 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:21.385381 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:21.385710 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48186: BlockingRecv error: recv got EOF from 127.0.0.1:48186 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260128 15:02:21.442937 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 16731
I20260128 15:02:21.447613 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 16867
I20260128 15:02:21.452222 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 17003
I20260128 15:02:21.456487 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 16635
2026-01-28T15:02:21Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3022 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:02:21 dist-test-slave-m28t krb5kdc[17155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:02:21 dist-test-slave-m28t krb5kdc[17155](info): set up 2 sockets
Jan 28 15:02:21 dist-test-slave-m28t krb5kdc[17155](info): commencing operation
krb5kdc: starting...
W20260128 15:02:23.387272 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.911s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260128 15:02:23.656050 13666 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.269s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260128 15:02:23.838016 13666 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.182s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260128 15:02:24.075781 13666 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.238s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/krb5kdc/test-user.keytab.
W20260128 15:02:25.018007 13666 mini_kdc.cc:290] Time spent creating keytab for test-user: real 0.942s user 0.000s sys 0.002s
Jan 28 15:02:25 dist-test-slave-m28t krb5kdc[17155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:02:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/krb5kdc/kudu.keytab.
W20260128 15:02:25.770251 13666 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.13.88.190: real 0.733s user 0.001s sys 0.001s
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:25.789078 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:44225
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:39437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:44225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:25.894410 17171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:25.894625 17171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:25.894680 17171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:25.897908 17171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:25.897960 17171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:25.897981 17171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:25.897997 17171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:25.898013 17171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:02:25.902210 17171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:39437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:44225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:44225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:25.903260 17171 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:25.904068 17171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:25.909502 17176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:25.909495 17179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:25.909554 17177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:25.909794 17171 server_base.cc:1060] running on GCE node
I20260128 15:02:25.910247 17171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:25.911135 17171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:25.912282 17171 hybrid_clock.cc:648] HybridClock initialized: now 1769612545912268 us; error 26 us; skew 500 ppm
Jan 28 15:02:25 dist-test-slave-m28t krb5kdc[17155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612545, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:25.914710 17171 init.cc:377] Logged in from keytab as kudu/127.13.88.190@KRBTEST.COM (short username kudu)
I20260128 15:02:25.915752 17171 webserver.cc:492] Webserver started at http://127.13.88.190:34341/ using document root <none> and password file <none>
I20260128 15:02:25.916261 17171 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:25.916304 17171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:25.916507 17171 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:25.918157 17171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "84bcfb60895c4c2abea9bfc90aa0e2bf"
format_stamp: "Formatted at 2026-01-28 15:02:25 on dist-test-slave-m28t"
server_key: "411c54e1e0c4ba555d081f1c204f67c2"
server_key_iv: "d861e3bf81305c8791b0710c693412b9"
server_key_version: "encryptionkey@0"
I20260128 15:02:25.918560 17171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "84bcfb60895c4c2abea9bfc90aa0e2bf"
format_stamp: "Formatted at 2026-01-28 15:02:25 on dist-test-slave-m28t"
server_key: "411c54e1e0c4ba555d081f1c204f67c2"
server_key_iv: "d861e3bf81305c8791b0710c693412b9"
server_key_version: "encryptionkey@0"
I20260128 15:02:25.921761 17171 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260128 15:02:25.924058 17186 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:25.925150 17171 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260128 15:02:25.925247 17171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "84bcfb60895c4c2abea9bfc90aa0e2bf"
format_stamp: "Formatted at 2026-01-28 15:02:25 on dist-test-slave-m28t"
server_key: "411c54e1e0c4ba555d081f1c204f67c2"
server_key_iv: "d861e3bf81305c8791b0710c693412b9"
server_key_version: "encryptionkey@0"
I20260128 15:02:25.925328 17171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:25.950650 17171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:25.957687 17171 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:25.957852 17171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:25.964947 17171 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:44225
I20260128 15:02:25.964982 17238 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:44225 every 8 connection(s)
I20260128 15:02:25.966008 17171 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:25.969071 17239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:25.973963 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 17171
I20260128 15:02:25.974057 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:25.974275 13666 external_mini_cluster.cc:1468] Setting key 6b367ecbcaee907f772235360a654de8
I20260128 15:02:25.974618 17239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf: Bootstrap starting.
I20260128 15:02:25.976926 17239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:25.977804 17239 log.cc:826] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:25.979697 17239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf: No bootstrap required, opened a new log
Jan 28 15:02:25 dist-test-slave-m28t krb5kdc[17155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:25.982554 17239 raft_consensus.cc:359] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bcfb60895c4c2abea9bfc90aa0e2bf" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44225 } }
I20260128 15:02:25.982734 17239 raft_consensus.cc:385] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:25.982784 17239 raft_consensus.cc:740] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84bcfb60895c4c2abea9bfc90aa0e2bf, State: Initialized, Role: FOLLOWER
I20260128 15:02:25.983181 17239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bcfb60895c4c2abea9bfc90aa0e2bf" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44225 } }
I20260128 15:02:25.983290 17239 raft_consensus.cc:399] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:25.983346 17239 raft_consensus.cc:493] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:25.983428 17239 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:25.984136 17239 raft_consensus.cc:515] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bcfb60895c4c2abea9bfc90aa0e2bf" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44225 } }
I20260128 15:02:25.984531 17239 leader_election.cc:304] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84bcfb60895c4c2abea9bfc90aa0e2bf; no voters:
I20260128 15:02:25.984848 17239 leader_election.cc:290] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:25.984933 17244 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:25.985437 17244 raft_consensus.cc:697] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [term 1 LEADER]: Becoming Leader. State: Replica: 84bcfb60895c4c2abea9bfc90aa0e2bf, State: Running, Role: LEADER
I20260128 15:02:25.985783 17244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bcfb60895c4c2abea9bfc90aa0e2bf" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44225 } }
I20260128 15:02:25.985986 17239 sys_catalog.cc:565] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:25.986572 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:25.975449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.13.88.190:44225)
0128 15:02:25.975870 (+ 421us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:25.975880 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:25.975914 (+ 34us) server_negotiation.cc:408] Connection header received
0128 15:02:25.976458 (+ 544us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:25.976479 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:25.976774 (+ 295us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:25.977062 (+ 288us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:25.977998 (+ 936us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:25.978918 (+ 920us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:25.979694 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:25.979937 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:25.982219 (+ 2282us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:25.982235 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:25.982245 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:25.982276 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:25.983976 (+ 1700us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:25.984391 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:25.984395 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:25.984399 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:25.984503 (+ 104us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:25.984722 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:25.984725 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:25.984727 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:25.985152 (+ 425us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:25.985364 (+ 212us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:25.985601 (+ 237us) server_negotiation.cc:300] Negotiation successful
0128 15:02:25.985853 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":142,"threads_started":1}
I20260128 15:02:25.986987 17244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84bcfb60895c4c2abea9bfc90aa0e2bf. Latest consensus state: current_term: 1 leader_uuid: "84bcfb60895c4c2abea9bfc90aa0e2bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bcfb60895c4c2abea9bfc90aa0e2bf" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44225 } } }
I20260128 15:02:25.987159 17244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:25.987295 17245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84bcfb60895c4c2abea9bfc90aa0e2bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84bcfb60895c4c2abea9bfc90aa0e2bf" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44225 } } }
I20260128 15:02:25.987385 17245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:25.988462 17253 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:25.990006 13666 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260128 15:02:25.991415 17253 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:25.998514 17253 catalog_manager.cc:1357] Generated new cluster ID: 386511e01bc841c28771bed8399ea18b
I20260128 15:02:25.998574 17253 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:26.015748 17253 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:26.017035 17253 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:26.017807 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:26.012950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.13.88.190:44225)
0128 15:02:26.013105 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:26.013110 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:26.013493 (+ 383us) server_negotiation.cc:408] Connection header received
0128 15:02:26.013630 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:26.013634 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:26.013692 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:26.013780 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:26.014541 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:26.015090 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:26.015893 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:26.016062 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:26.016847 (+ 785us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:26.016864 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:26.016868 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:02:26.017351 (+ 483us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:26.017388 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:02:26.017406 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:26.017489 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:26.017590 (+ 101us) server_negotiation.cc:300] Negotiation successful
0128 15:02:26.017662 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260128 15:02:26.031147 17253 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 84bcfb60895c4c2abea9bfc90aa0e2bf: Generated new TSK 0
I20260128 15:02:26.031890 17253 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260128 15:02:26.039433 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 17171
2026-01-28T15:02:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (4581 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:02:26 dist-test-slave-m28t krb5kdc[17272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:02:26 dist-test-slave-m28t krb5kdc[17272](info): set up 2 sockets
Jan 28 15:02:26 dist-test-slave-m28t krb5kdc[17272](info): commencing operation
krb5kdc: starting...
W20260128 15:02:27.864367 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.806s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Jan 28 15:02:27 dist-test-slave-m28t krb5kdc[17272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:02:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:27.988922 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:44389
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:34157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:44389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:28.091372 17288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:28.091611 17288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:28.091682 17288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:28.094959 17288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:28.095026 17288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:28.095057 17288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:28.095082 17288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:28.095113 17288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:02:28.099450 17288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:34157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:44389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:44389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:28.100651 17288 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:28.101516 17288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:28.106995 17293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:28.106990 17296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:28.107129 17288 server_base.cc:1060] running on GCE node
W20260128 15:02:28.106992 17294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:28.107718 17288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:28.108630 17288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:28.109841 17288 hybrid_clock.cc:648] HybridClock initialized: now 1769612548109830 us; error 37 us; skew 500 ppm
Jan 28 15:02:28 dist-test-slave-m28t krb5kdc[17272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612548, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:28.112404 17288 init.cc:377] Logged in from keytab as kudu/127.13.88.190@KRBTEST.COM (short username kudu)
I20260128 15:02:28.113446 17288 webserver.cc:492] Webserver started at http://127.13.88.190:38641/ using document root <none> and password file <none>
I20260128 15:02:28.113955 17288 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:28.114001 17288 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:28.114166 17288 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:28.115850 17288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "2699fd02cdd547059c7a0d2dee21f250"
format_stamp: "Formatted at 2026-01-28 15:02:28 on dist-test-slave-m28t"
server_key: "fa29ac2cf31751d43aff7424fabc21c1"
server_key_iv: "881146424ef11d53b006c78de2c840fb"
server_key_version: "encryptionkey@0"
I20260128 15:02:28.116302 17288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "2699fd02cdd547059c7a0d2dee21f250"
format_stamp: "Formatted at 2026-01-28 15:02:28 on dist-test-slave-m28t"
server_key: "fa29ac2cf31751d43aff7424fabc21c1"
server_key_iv: "881146424ef11d53b006c78de2c840fb"
server_key_version: "encryptionkey@0"
I20260128 15:02:28.119491 17288 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260128 15:02:28.121549 17303 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:28.122480 17288 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:02:28.122591 17288 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "2699fd02cdd547059c7a0d2dee21f250"
format_stamp: "Formatted at 2026-01-28 15:02:28 on dist-test-slave-m28t"
server_key: "fa29ac2cf31751d43aff7424fabc21c1"
server_key_iv: "881146424ef11d53b006c78de2c840fb"
server_key_version: "encryptionkey@0"
I20260128 15:02:28.122671 17288 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:28.134250 17288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:28.136897 17288 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:28.137037 17288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:28.143208 17355 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:44389 every 8 connection(s)
I20260128 15:02:28.143210 17288 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:44389
I20260128 15:02:28.144176 17288 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:28.146767 17356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:28.151973 17356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250: Bootstrap starting.
I20260128 15:02:28.153223 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 17288
I20260128 15:02:28.153328 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:28.153543 13666 external_mini_cluster.cc:1468] Setting key d0038606d93d7bfe10d55e0ed0960beb
I20260128 15:02:28.154234 17356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:28.155055 17356 log.cc:826] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:28.156911 17356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250: No bootstrap required, opened a new log
I20260128 15:02:28.159662 17356 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2699fd02cdd547059c7a0d2dee21f250" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44389 } }
I20260128 15:02:28.159879 17356 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:28.159992 17356 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2699fd02cdd547059c7a0d2dee21f250, State: Initialized, Role: FOLLOWER
Jan 28 15:02:28 dist-test-slave-m28t krb5kdc[17272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:28.160408 17356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2699fd02cdd547059c7a0d2dee21f250" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44389 } }
I20260128 15:02:28.160569 17356 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:28.160630 17356 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:28.160718 17356 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:28.161638 17356 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2699fd02cdd547059c7a0d2dee21f250" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44389 } }
I20260128 15:02:28.162006 17356 leader_election.cc:304] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2699fd02cdd547059c7a0d2dee21f250; no voters:
I20260128 15:02:28.162317 17356 leader_election.cc:290] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:28.162644 17361 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:28.162864 17361 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [term 1 LEADER]: Becoming Leader. State: Replica: 2699fd02cdd547059c7a0d2dee21f250, State: Running, Role: LEADER
I20260128 15:02:28.163147 17361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2699fd02cdd547059c7a0d2dee21f250" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44389 } }
I20260128 15:02:28.163429 17356 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:28.164387 17361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2699fd02cdd547059c7a0d2dee21f250. Latest consensus state: current_term: 1 leader_uuid: "2699fd02cdd547059c7a0d2dee21f250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2699fd02cdd547059c7a0d2dee21f250" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44389 } } }
I20260128 15:02:28.164598 17361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:28.164733 17362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2699fd02cdd547059c7a0d2dee21f250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2699fd02cdd547059c7a0d2dee21f250" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44389 } } }
I20260128 15:02:28.164812 17362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250 [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:28.165108 17370 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:28.166318 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:28.154437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.13.88.190:44389)
0128 15:02:28.154788 (+ 351us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:28.154798 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:28.154831 (+ 33us) server_negotiation.cc:408] Connection header received
0128 15:02:28.155356 (+ 525us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:28.155374 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:28.155664 (+ 290us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:28.155919 (+ 255us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:28.156891 (+ 972us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:28.157601 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:28.158440 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:28.158656 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:28.161009 (+ 2353us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:28.161029 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:28.161041 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:28.161081 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:28.163083 (+ 2002us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:28.163574 (+ 491us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:28.163579 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:28.163585 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:28.163677 (+ 92us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:28.163946 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:28.163950 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:28.163952 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:28.164390 (+ 438us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:28.165177 (+ 787us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:28.165353 (+ 176us) server_negotiation.cc:300] Negotiation successful
0128 15:02:28.165614 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20260128 15:02:28.169103 17370 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:28.173663 17370 catalog_manager.cc:1357] Generated new cluster ID: e7ce95b712374253913e915b6e4182eb
I20260128 15:02:28.173735 17370 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:28.189513 17370 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:28.190701 17370 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:28.203094 17370 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 2699fd02cdd547059c7a0d2dee21f250: Generated new TSK 0
I20260128 15:02:28.203841 17370 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:28.250515 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:44389
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:34157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:28.357225 17384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:28.357484 17384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:28.357543 17384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:28.360884 17384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:28.360951 17384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:28.361033 17384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:02:28.365516 17384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:34157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:44389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:28.366557 17384 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:28.367377 17384 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:28.373713 17392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:28.373713 17390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:28.373760 17389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:28.374104 17384 server_base.cc:1060] running on GCE node
I20260128 15:02:28.374454 17384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:28.374984 17384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:28.376130 17384 hybrid_clock.cc:648] HybridClock initialized: now 1769612548376119 us; error 26 us; skew 500 ppm
Jan 28 15:02:28 dist-test-slave-m28t krb5kdc[17272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612548, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:28.378648 17384 init.cc:377] Logged in from keytab as kudu/127.13.88.129@KRBTEST.COM (short username kudu)
I20260128 15:02:28.379722 17384 webserver.cc:492] Webserver started at http://127.13.88.129:35521/ using document root <none> and password file <none>
I20260128 15:02:28.380275 17384 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:28.380324 17384 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:28.380535 17384 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:28.382256 17384 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "d6460080a57644dca4300fda72ef2d23"
format_stamp: "Formatted at 2026-01-28 15:02:28 on dist-test-slave-m28t"
server_key: "32fe2683a88692f6c20c45031b0289a1"
server_key_iv: "c2729bbb2a8e2547c8a63005a7515a6e"
server_key_version: "encryptionkey@0"
I20260128 15:02:28.382694 17384 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "d6460080a57644dca4300fda72ef2d23"
format_stamp: "Formatted at 2026-01-28 15:02:28 on dist-test-slave-m28t"
server_key: "32fe2683a88692f6c20c45031b0289a1"
server_key_iv: "c2729bbb2a8e2547c8a63005a7515a6e"
server_key_version: "encryptionkey@0"
I20260128 15:02:28.386142 17384 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260128 15:02:28.388386 17399 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:28.389452 17384 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260128 15:02:28.389552 17384 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "d6460080a57644dca4300fda72ef2d23"
format_stamp: "Formatted at 2026-01-28 15:02:28 on dist-test-slave-m28t"
server_key: "32fe2683a88692f6c20c45031b0289a1"
server_key_iv: "c2729bbb2a8e2547c8a63005a7515a6e"
server_key_version: "encryptionkey@0"
I20260128 15:02:28.389638 17384 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:28.423372 17384 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:28.426441 17384 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:28.426625 17384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:28.427208 17384 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:28.428126 17384 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:28.428174 17384 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:28.428225 17384 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:28.428252 17384 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:28.437065 17384 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:43139
I20260128 15:02:28.437086 17512 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:43139 every 8 connection(s)
I20260128 15:02:28.438004 17384 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:02:28.446108 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 17384
I20260128 15:02:28.446216 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
Jan 28 15:02:28 dist-test-slave-m28t krb5kdc[17272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612548, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:28.446447 13666 external_mini_cluster.cc:1468] Setting key 18d40ca982acb8dce8266f293128a38b
I20260128 15:02:28.450212 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:28.439847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:41293 (local address 127.13.88.190:44389)
0128 15:02:28.439992 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:28.439996 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:28.440729 (+ 733us) server_negotiation.cc:408] Connection header received
0128 15:02:28.441602 (+ 873us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:28.441606 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:28.441670 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:28.441747 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:28.443431 (+ 1684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:28.443949 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:28.444613 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:28.444802 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:28.447034 (+ 2232us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:28.447052 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:28.447054 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:28.447080 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:28.448397 (+ 1317us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:28.448888 (+ 491us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:28.448891 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:28.448893 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:28.448945 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:28.449194 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:28.449197 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:28.449199 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:28.449347 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:28.449429 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:28.449920 (+ 491us) server_negotiation.cc:300] Negotiation successful
0128 15:02:28.450058 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260128 15:02:28.450836 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:28.440120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:44389 (local address 127.13.88.129:41293)
0128 15:02:28.440575 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:28.440613 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:28.441401 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:28.441932 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:28.441941 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:28.442335 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:28.443270 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:28.443285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:28.444069 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:28.444072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:28.444503 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:28.444511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:28.444740 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:28.445336 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:28.445363 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:28.446841 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:28.448534 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:28.448539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:28.448552 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:28.448795 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:28.449022 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:28.449025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:28.449026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:28.449114 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:28.449441 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:28.449448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:28.449692 (+ 244us) client_negotiation.cc:769] Sending connection context
0128 15:02:28.449876 (+ 184us) client_negotiation.cc:240] Negotiation successful
0128 15:02:28.450144 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
I20260128 15:02:28.452018 17513 heartbeater.cc:344] Connected to a master server at 127.13.88.190:44389
I20260128 15:02:28.452246 17513 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:28.452801 17513 heartbeater.cc:507] Master 127.13.88.190:44389 requested a full tablet report, sending...
I20260128 15:02:28.454216 17319 ts_manager.cc:194] Registered new tserver with Master: d6460080a57644dca4300fda72ef2d23 (127.13.88.129:43139)
I20260128 15:02:28.455402 17319 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.129@KRBTEST.COM'} at 127.13.88.129:41293
WARNING: no policy specified for kudu/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:28.491518 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:44389
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:34157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:28.593924 17520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:28.594146 17520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:28.594203 17520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:28.597458 17520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:28.597517 17520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:28.597594 17520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:02:28.601619 17520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:34157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:44389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:28.602582 17520 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:28.603374 17520 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:28.609618 17525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:28.609738 17528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:28.609789 17520 server_base.cc:1060] running on GCE node
W20260128 15:02:28.609906 17526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:28.610242 17520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:28.610772 17520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:28.611913 17520 hybrid_clock.cc:648] HybridClock initialized: now 1769612548611903 us; error 24 us; skew 500 ppm
Jan 28 15:02:28 dist-test-slave-m28t krb5kdc[17272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612548, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:28.614270 17520 init.cc:377] Logged in from keytab as kudu/127.13.88.130@KRBTEST.COM (short username kudu)
I20260128 15:02:28.615239 17520 webserver.cc:492] Webserver started at http://127.13.88.130:38903/ using document root <none> and password file <none>
I20260128 15:02:28.615753 17520 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:28.615811 17520 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:28.615985 17520 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:28.617597 17520 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "f5b3cf7d94a540b19fda1af04fc365b5"
format_stamp: "Formatted at 2026-01-28 15:02:28 on dist-test-slave-m28t"
server_key: "04bb9c8c8b1f383bbbe8eb4f67b77e91"
server_key_iv: "be4f5cd7787e20ba19645513123234b6"
server_key_version: "encryptionkey@0"
I20260128 15:02:28.618021 17520 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "f5b3cf7d94a540b19fda1af04fc365b5"
format_stamp: "Formatted at 2026-01-28 15:02:28 on dist-test-slave-m28t"
server_key: "04bb9c8c8b1f383bbbe8eb4f67b77e91"
server_key_iv: "be4f5cd7787e20ba19645513123234b6"
server_key_version: "encryptionkey@0"
I20260128 15:02:28.621254 17520 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260128 15:02:28.623302 17535 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:28.624226 17520 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:28.624320 17520 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "f5b3cf7d94a540b19fda1af04fc365b5"
format_stamp: "Formatted at 2026-01-28 15:02:28 on dist-test-slave-m28t"
server_key: "04bb9c8c8b1f383bbbe8eb4f67b77e91"
server_key_iv: "be4f5cd7787e20ba19645513123234b6"
server_key_version: "encryptionkey@0"
I20260128 15:02:28.624403 17520 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:28.642470 17520 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:28.645038 17520 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:28.645218 17520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:28.645750 17520 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:28.646625 17520 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:28.646672 17520 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:28.646731 17520 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:28.646757 17520 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:28.654992 17520 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:46115
I20260128 15:02:28.655020 17648 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:46115 every 8 connection(s)
I20260128 15:02:28.655905 17520 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
I20260128 15:02:28.656612 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 17520
I20260128 15:02:28.656718 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:02:28.656944 13666 external_mini_cluster.cc:1468] Setting key 2e91b6a6a135121191c2c1654d9d54bb
Jan 28 15:02:28 dist-test-slave-m28t krb5kdc[17272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612548, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.131@KRBTEST.COM" created.
I20260128 15:02:28.668833 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:28.657592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:52871 (local address 127.13.88.190:44389)
0128 15:02:28.657724 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:28.657728 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:28.658546 (+ 818us) server_negotiation.cc:408] Connection header received
0128 15:02:28.659386 (+ 840us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:28.659390 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:28.659445 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:28.659528 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:28.661437 (+ 1909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:28.662223 (+ 786us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:28.662810 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:28.663020 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:28.664978 (+ 1958us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:28.665000 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:28.665019 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:28.665052 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:28.666307 (+ 1255us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:28.666919 (+ 612us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:28.666923 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:28.666926 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:28.666983 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:28.667458 (+ 475us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:28.667461 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:28.667463 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:28.667752 (+ 289us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:28.667833 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:28.668536 (+ 703us) server_negotiation.cc:300] Negotiation successful
0128 15:02:28.668666 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260128 15:02:28.669658 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:28.657937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:44389 (local address 127.13.88.130:52871)
0128 15:02:28.658411 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:28.658452 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:28.659199 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:28.659681 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:28.659692 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:28.660231 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:28.661276 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:28.661295 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:28.662329 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:28.662331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:28.662711 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:28.662717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:28.662875 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:28.663389 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:28.663413 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:28.664817 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:28.666422 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:28.666427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:28.666438 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:28.666811 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:28.667098 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:28.667102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:28.667104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:28.667292 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:28.667914 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:28.667921 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:28.668226 (+ 305us) client_negotiation.cc:769] Sending connection context
0128 15:02:28.668494 (+ 268us) client_negotiation.cc:240] Negotiation successful
0128 15:02:28.668780 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":129,"threads_started":1}
I20260128 15:02:28.670710 17649 heartbeater.cc:344] Connected to a master server at 127.13.88.190:44389
I20260128 15:02:28.670979 17649 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:28.671607 17649 heartbeater.cc:507] Master 127.13.88.190:44389 requested a full tablet report, sending...
I20260128 15:02:28.672676 17319 ts_manager.cc:194] Registered new tserver with Master: f5b3cf7d94a540b19fda1af04fc365b5 (127.13.88.130:46115)
I20260128 15:02:28.673233 17319 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.130@KRBTEST.COM'} at 127.13.88.130:52871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:28.700206 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:44389
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:34157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:28.803854 17656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:28.804109 17656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:28.804199 17656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:28.807566 17656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:28.807646 17656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:28.807751 17656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:02:28.812137 17656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:34157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:44389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:28.813196 17656 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:28.814009 17656 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:28.820403 17664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:28.820397 17661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:28.820397 17662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:28.820559 17656 server_base.cc:1060] running on GCE node
I20260128 15:02:28.821033 17656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:28.821568 17656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:28.822721 17656 hybrid_clock.cc:648] HybridClock initialized: now 1769612548822710 us; error 31 us; skew 500 ppm
Jan 28 15:02:28 dist-test-slave-m28t krb5kdc[17272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612548, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:28.825271 17656 init.cc:377] Logged in from keytab as kudu/127.13.88.131@KRBTEST.COM (short username kudu)
I20260128 15:02:28.826247 17656 webserver.cc:492] Webserver started at http://127.13.88.131:36163/ using document root <none> and password file <none>
I20260128 15:02:28.826799 17656 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:28.826848 17656 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:28.827006 17656 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:28.828644 17656 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "573b14361a8a460ca61d2e31783e3847"
format_stamp: "Formatted at 2026-01-28 15:02:28 on dist-test-slave-m28t"
server_key: "2a35da26f0018ea9a610928936e47e94"
server_key_iv: "190a4595a276ce0a414a3c61eb9b56b3"
server_key_version: "encryptionkey@0"
I20260128 15:02:28.829109 17656 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "573b14361a8a460ca61d2e31783e3847"
format_stamp: "Formatted at 2026-01-28 15:02:28 on dist-test-slave-m28t"
server_key: "2a35da26f0018ea9a610928936e47e94"
server_key_iv: "190a4595a276ce0a414a3c61eb9b56b3"
server_key_version: "encryptionkey@0"
I20260128 15:02:28.832227 17656 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260128 15:02:28.834368 17671 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:28.835338 17656 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260128 15:02:28.835428 17656 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "573b14361a8a460ca61d2e31783e3847"
format_stamp: "Formatted at 2026-01-28 15:02:28 on dist-test-slave-m28t"
server_key: "2a35da26f0018ea9a610928936e47e94"
server_key_iv: "190a4595a276ce0a414a3c61eb9b56b3"
server_key_version: "encryptionkey@0"
I20260128 15:02:28.835501 17656 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:28.850049 17656 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:28.852878 17656 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:28.853022 17656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:28.853487 17656 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:28.854322 17656 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:28.854368 17656 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:28.854405 17656 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:28.854425 17656 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:28.862670 17656 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:39035
I20260128 15:02:28.862702 17784 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:39035 every 8 connection(s)
I20260128 15:02:28.863587 17656 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
I20260128 15:02:28.864957 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 17656
I20260128 15:02:28.865092 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:02:28.865366 13666 external_mini_cluster.cc:1468] Setting key 001ff00cda2ba4838c3ab8a31cce54be
Jan 28 15:02:28 dist-test-slave-m28t krb5kdc[17272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612548, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:28.875313 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:28.865294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:37935 (local address 127.13.88.190:44389)
0128 15:02:28.865418 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:28.865423 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:28.866391 (+ 968us) server_negotiation.cc:408] Connection header received
0128 15:02:28.867342 (+ 951us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:28.867345 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:28.867399 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:28.867466 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:28.869098 (+ 1632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:28.869582 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:28.870167 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:28.870344 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:28.872346 (+ 2002us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:28.872363 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:28.872365 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:28.872391 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:28.873558 (+ 1167us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:28.874018 (+ 460us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:28.874021 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:28.874022 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:28.874065 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:28.874310 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:28.874313 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:28.874315 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:28.874466 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:28.874550 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:28.875039 (+ 489us) server_negotiation.cc:300] Negotiation successful
0128 15:02:28.875155 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260128 15:02:28.875919 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:28.865651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:44389 (local address 127.13.88.131:37935)
0128 15:02:28.866258 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:28.866300 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:28.867145 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:28.867641 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:28.867649 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:28.868019 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:28.868965 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:28.868979 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:28.869686 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:28.869689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:28.870065 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:28.870071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:28.870267 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:28.870800 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:28.870823 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:28.872203 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:28.873680 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:28.873686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:28.873698 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:28.873924 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:28.874140 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:28.874143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:28.874145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:28.874231 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:28.874559 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:28.874566 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:28.874807 (+ 241us) client_negotiation.cc:769] Sending connection context
0128 15:02:28.874990 (+ 183us) client_negotiation.cc:240] Negotiation successful
0128 15:02:28.875242 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260128 15:02:28.876982 17785 heartbeater.cc:344] Connected to a master server at 127.13.88.190:44389
I20260128 15:02:28.877215 17785 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:28.877756 17785 heartbeater.cc:507] Master 127.13.88.190:44389 requested a full tablet report, sending...
I20260128 15:02:28.878785 17320 ts_manager.cc:194] Registered new tserver with Master: 573b14361a8a460ca61d2e31783e3847 (127.13.88.131:39035)
I20260128 15:02:28.879369 17320 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.131@KRBTEST.COM'} at 127.13.88.131:37935
I20260128 15:02:28.890470 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 28 15:02:28 dist-test-slave-m28t krb5kdc[17272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612548, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jan 28 15:02:28 dist-test-slave-m28t krb5kdc[17272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612548, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:28.911579 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:28.904965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58614 (local address 127.13.88.190:44389)
0128 15:02:28.905080 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:28.905083 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:28.905168 (+ 85us) server_negotiation.cc:408] Connection header received
0128 15:02:28.905292 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:28.905295 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:28.905339 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:28.905398 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:28.906104 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:28.906588 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:28.907169 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:28.907297 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:28.909246 (+ 1949us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:28.909265 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:28.909267 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:28.909292 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:28.910450 (+ 1158us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:28.910775 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:28.910778 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:28.910779 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:28.910813 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:28.910963 (+ 150us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:28.910965 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:28.910966 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:28.911131 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:28.911224 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:28.911370 (+ 146us) server_negotiation.cc:300] Negotiation successful
0128 15:02:28.911458 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260128 15:02:28.911960 17320 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58614
I20260128 15:02:28.912761 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 17384
I20260128 15:02:28.917498 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 17520
I20260128 15:02:28.922140 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 17656
I20260128 15:02:28.926621 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 17288
2026-01-28T15:02:28Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2888 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:02:28 dist-test-slave-m28t krb5kdc[17797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:02:28 dist-test-slave-m28t krb5kdc[17797](info): set up 2 sockets
Jan 28 15:02:28 dist-test-slave-m28t krb5kdc[17797](info): commencing operation
krb5kdc: starting...
W20260128 15:02:30.760520 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.814s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Jan 28 15:02:30 dist-test-slave-m28t krb5kdc[17797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:02:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:30.884783 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:44507
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:45653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:44507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:30.990248 17813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:30.990501 17813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:30.990581 17813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:30.994035 17813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:30.994119 17813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:30.994153 17813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:30.994177 17813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:30.994202 17813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:02:30.998829 17813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:45653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:44507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:44507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:31.000012 17813 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:31.000972 17813 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:31.007053 17818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:31.007236 17813 server_base.cc:1060] running on GCE node
W20260128 15:02:31.007053 17819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:31.007053 17821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:31.007880 17813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:31.008905 17813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:31.010057 17813 hybrid_clock.cc:648] HybridClock initialized: now 1769612551010034 us; error 43 us; skew 500 ppm
Jan 28 15:02:31 dist-test-slave-m28t krb5kdc[17797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612551, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:31.012699 17813 init.cc:377] Logged in from keytab as kudu/127.13.88.190@KRBTEST.COM (short username kudu)
I20260128 15:02:31.013912 17813 webserver.cc:492] Webserver started at http://127.13.88.190:40531/ using document root <none> and password file <none>
I20260128 15:02:31.014503 17813 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:31.014554 17813 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:31.014752 17813 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:31.016753 17813 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "7b634e881b694de09c4410c43302c807"
format_stamp: "Formatted at 2026-01-28 15:02:31 on dist-test-slave-m28t"
server_key: "81ad411e8bc26174c72fa308ac31aae0"
server_key_iv: "c3bbc63d5eebabc9f1f3e5513a2527d8"
server_key_version: "encryptionkey@0"
I20260128 15:02:31.017252 17813 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "7b634e881b694de09c4410c43302c807"
format_stamp: "Formatted at 2026-01-28 15:02:31 on dist-test-slave-m28t"
server_key: "81ad411e8bc26174c72fa308ac31aae0"
server_key_iv: "c3bbc63d5eebabc9f1f3e5513a2527d8"
server_key_version: "encryptionkey@0"
I20260128 15:02:31.020738 17813 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260128 15:02:31.022842 17828 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:31.023851 17813 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:31.023962 17813 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "7b634e881b694de09c4410c43302c807"
format_stamp: "Formatted at 2026-01-28 15:02:31 on dist-test-slave-m28t"
server_key: "81ad411e8bc26174c72fa308ac31aae0"
server_key_iv: "c3bbc63d5eebabc9f1f3e5513a2527d8"
server_key_version: "encryptionkey@0"
I20260128 15:02:31.024051 17813 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:31.038105 17813 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:31.044987 17813 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:31.045157 17813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:31.051834 17880 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:44507 every 8 connection(s)
I20260128 15:02:31.051841 17813 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:44507
I20260128 15:02:31.052923 17813 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:31.055595 17881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:31.059908 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 17813
I20260128 15:02:31.060031 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:31.060268 13666 external_mini_cluster.cc:1468] Setting key ab876b34a1e84b5eed058922861b80ca
I20260128 15:02:31.061388 17881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807: Bootstrap starting.
I20260128 15:02:31.063835 17881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:31.064612 17881 log.cc:826] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:31.066578 17881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807: No bootstrap required, opened a new log
Jan 28 15:02:31 dist-test-slave-m28t krb5kdc[17797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:31.069382 17881 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b634e881b694de09c4410c43302c807" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44507 } }
I20260128 15:02:31.069571 17881 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:31.069641 17881 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b634e881b694de09c4410c43302c807, State: Initialized, Role: FOLLOWER
I20260128 15:02:31.070053 17881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b634e881b694de09c4410c43302c807" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44507 } }
I20260128 15:02:31.070178 17881 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:31.070238 17881 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:31.070322 17881 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:31.071311 17881 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b634e881b694de09c4410c43302c807" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44507 } }
I20260128 15:02:31.071702 17881 leader_election.cc:304] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b634e881b694de09c4410c43302c807; no voters:
I20260128 15:02:31.072021 17881 leader_election.cc:290] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:31.072593 17887 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:31.072834 17887 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [term 1 LEADER]: Becoming Leader. State: Replica: 7b634e881b694de09c4410c43302c807, State: Running, Role: LEADER
I20260128 15:02:31.072989 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:31.061529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.13.88.190:44507)
0128 15:02:31.061919 (+ 390us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:31.061928 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:31.061964 (+ 36us) server_negotiation.cc:408] Connection header received
0128 15:02:31.062489 (+ 525us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:31.062506 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:31.062822 (+ 316us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:31.063095 (+ 273us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:31.064095 (+ 1000us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.064963 (+ 868us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:31.065709 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.065917 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:31.068312 (+ 2395us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:31.068331 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:31.068350 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:31.068387 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:31.070167 (+ 1780us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:31.070623 (+ 456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:31.070629 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:31.070634 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:31.070742 (+ 108us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:31.070953 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:31.070956 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:31.070958 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:31.071420 (+ 462us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:31.071624 (+ 204us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:31.071928 (+ 304us) server_negotiation.cc:300] Negotiation successful
0128 15:02:31.072213 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
I20260128 15:02:31.073187 17881 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:31.074013 17887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b634e881b694de09c4410c43302c807" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44507 } }
I20260128 15:02:31.075693 17886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b634e881b694de09c4410c43302c807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b634e881b694de09c4410c43302c807" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44507 } } }
I20260128 15:02:31.076045 17886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:31.075623 17888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b634e881b694de09c4410c43302c807. Latest consensus state: current_term: 1 leader_uuid: "7b634e881b694de09c4410c43302c807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b634e881b694de09c4410c43302c807" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 44507 } } }
I20260128 15:02:31.076372 17888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807 [sys.catalog]: This master's current role is: LEADER
W20260128 15:02:31.077924 17900 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 15:02:31.077991 17900 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 15:02:31.078047 17901 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:31.078697 17901 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:31.083525 17901 catalog_manager.cc:1357] Generated new cluster ID: f6f71798613544b99791f7322766c9aa
I20260128 15:02:31.083606 17901 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:31.101961 17901 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:31.103261 17901 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:31.113783 17901 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 7b634e881b694de09c4410c43302c807: Generated new TSK 0
I20260128 15:02:31.114616 17901 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:31.161403 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:44507
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:45653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:31.296506 17909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:31.296794 17909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:31.296878 17909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:31.300186 17909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:31.300248 17909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:31.300326 17909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:02:31.304471 17909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:45653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:44507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:31.305522 17909 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:31.306329 17909 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:31.312757 17917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:31.312757 17915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:31.312836 17914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:31.312932 17909 server_base.cc:1060] running on GCE node
I20260128 15:02:31.313300 17909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:31.313825 17909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:31.314967 17909 hybrid_clock.cc:648] HybridClock initialized: now 1769612551314945 us; error 38 us; skew 500 ppm
Jan 28 15:02:31 dist-test-slave-m28t krb5kdc[17797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612551, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:31.317493 17909 init.cc:377] Logged in from keytab as kudu/127.13.88.129@KRBTEST.COM (short username kudu)
I20260128 15:02:31.318526 17909 webserver.cc:492] Webserver started at http://127.13.88.129:41557/ using document root <none> and password file <none>
I20260128 15:02:31.319052 17909 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:31.319103 17909 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:31.319271 17909 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:31.320958 17909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "93a5f2f4a7344bd692805c0c67446aad"
format_stamp: "Formatted at 2026-01-28 15:02:31 on dist-test-slave-m28t"
server_key: "413f657bc63272957a5701ea401be777"
server_key_iv: "fb08444897e498e98116080edb5b5a88"
server_key_version: "encryptionkey@0"
I20260128 15:02:31.321398 17909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "93a5f2f4a7344bd692805c0c67446aad"
format_stamp: "Formatted at 2026-01-28 15:02:31 on dist-test-slave-m28t"
server_key: "413f657bc63272957a5701ea401be777"
server_key_iv: "fb08444897e498e98116080edb5b5a88"
server_key_version: "encryptionkey@0"
I20260128 15:02:31.324579 17909 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260128 15:02:31.326601 17924 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:31.327556 17909 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:31.327654 17909 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "93a5f2f4a7344bd692805c0c67446aad"
format_stamp: "Formatted at 2026-01-28 15:02:31 on dist-test-slave-m28t"
server_key: "413f657bc63272957a5701ea401be777"
server_key_iv: "fb08444897e498e98116080edb5b5a88"
server_key_version: "encryptionkey@0"
I20260128 15:02:31.327732 17909 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:31.342661 17909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:31.345487 17909 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:31.345635 17909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:31.346140 17909 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:31.347080 17909 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:31.347174 17909 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:31.347232 17909 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:31.347262 17909 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:31.356187 17909 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:34867
I20260128 15:02:31.356225 18037 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:34867 every 8 connection(s)
I20260128 15:02:31.357172 17909 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
Jan 28 15:02:31 dist-test-slave-m28t krb5kdc[17797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612551, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:31.367341 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 17909
I20260128 15:02:31.367463 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:02:31.367744 13666 external_mini_cluster.cc:1468] Setting key 6b154f51ec1858bf507d2bc06a31cd5d
I20260128 15:02:31.368862 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:31.358977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:37949 (local address 127.13.88.190:44507)
0128 15:02:31.359129 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:31.359134 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:31.359835 (+ 701us) server_negotiation.cc:408] Connection header received
0128 15:02:31.360718 (+ 883us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:31.360721 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:31.360776 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:31.360849 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:31.362451 (+ 1602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.362951 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:31.363539 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.363660 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:31.365711 (+ 2051us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:31.365733 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:31.365744 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:31.365770 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:31.367000 (+ 1230us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:31.367516 (+ 516us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:31.367519 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:31.367521 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:31.367569 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:31.367875 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:31.367878 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:31.367880 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:31.368032 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:31.368107 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:31.368636 (+ 529us) server_negotiation.cc:300] Negotiation successful
0128 15:02:31.368743 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260128 15:02:31.369626 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:31.359243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:44507 (local address 127.13.88.129:37949)
0128 15:02:31.359691 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:31.359730 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:31.360526 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:31.360997 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:31.361005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:31.361391 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:31.362329 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:31.362343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.363051 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:31.363054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:31.363455 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:31.363462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.363628 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:31.364148 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:31.364171 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:31.365561 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:31.367100 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:31.367105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:31.367118 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:31.367422 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:31.367651 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:31.367655 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:31.367656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:31.367784 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:31.368134 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:31.368140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:31.368369 (+ 229us) client_negotiation.cc:769] Sending connection context
0128 15:02:31.368604 (+ 235us) client_negotiation.cc:240] Negotiation successful
0128 15:02:31.368871 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":108,"threads_started":1}
I20260128 15:02:31.370748 18038 heartbeater.cc:344] Connected to a master server at 127.13.88.190:44507
I20260128 15:02:31.371026 18038 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:31.371601 18038 heartbeater.cc:507] Master 127.13.88.190:44507 requested a full tablet report, sending...
I20260128 15:02:31.373087 17845 ts_manager.cc:194] Registered new tserver with Master: 93a5f2f4a7344bd692805c0c67446aad (127.13.88.129:34867)
I20260128 15:02:31.374209 17845 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.129@KRBTEST.COM'} at 127.13.88.129:37949
WARNING: no policy specified for kudu/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:31.409400 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:44507
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:45653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:31.512271 18045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:31.512499 18045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:31.512573 18045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:31.515769 18045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:31.515829 18045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:31.515923 18045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:02:31.520012 18045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:45653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:44507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:31.521046 18045 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:31.521849 18045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:31.527865 18051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:31.527869 18053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:31.527906 18050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:31.528066 18045 server_base.cc:1060] running on GCE node
I20260128 15:02:31.528487 18045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:31.529007 18045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:31.530162 18045 hybrid_clock.cc:648] HybridClock initialized: now 1769612551530139 us; error 36 us; skew 500 ppm
Jan 28 15:02:31 dist-test-slave-m28t krb5kdc[17797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612551, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:31.532613 18045 init.cc:377] Logged in from keytab as kudu/127.13.88.130@KRBTEST.COM (short username kudu)
I20260128 15:02:31.533574 18045 webserver.cc:492] Webserver started at http://127.13.88.130:35111/ using document root <none> and password file <none>
I20260128 15:02:31.534063 18045 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:31.534104 18045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:31.534245 18045 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:31.535880 18045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "d835d9d5f725430293f991eca1915290"
format_stamp: "Formatted at 2026-01-28 15:02:31 on dist-test-slave-m28t"
server_key: "5fc7a5bc85d439cc890a6fc6fcea638b"
server_key_iv: "157819c5ace1fcc70bf28bc02ecbbcae"
server_key_version: "encryptionkey@0"
I20260128 15:02:31.536294 18045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "d835d9d5f725430293f991eca1915290"
format_stamp: "Formatted at 2026-01-28 15:02:31 on dist-test-slave-m28t"
server_key: "5fc7a5bc85d439cc890a6fc6fcea638b"
server_key_iv: "157819c5ace1fcc70bf28bc02ecbbcae"
server_key_version: "encryptionkey@0"
I20260128 15:02:31.539522 18045 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260128 15:02:31.541576 18060 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:31.542538 18045 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:02:31.542625 18045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "d835d9d5f725430293f991eca1915290"
format_stamp: "Formatted at 2026-01-28 15:02:31 on dist-test-slave-m28t"
server_key: "5fc7a5bc85d439cc890a6fc6fcea638b"
server_key_iv: "157819c5ace1fcc70bf28bc02ecbbcae"
server_key_version: "encryptionkey@0"
I20260128 15:02:31.542694 18045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:31.556408 18045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:31.558884 18045 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:31.559027 18045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:31.559489 18045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:31.560292 18045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:31.560335 18045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:31.560372 18045 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:31.560392 18045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:31.568701 18045 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:38493
I20260128 15:02:31.568717 18173 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:38493 every 8 connection(s)
I20260128 15:02:31.569621 18045 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
I20260128 15:02:31.574258 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 18045
I20260128 15:02:31.574355 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:02:31.574579 13666 external_mini_cluster.cc:1468] Setting key 75ed8f96affe13e6a32045ecd6c049a1
Jan 28 15:02:31 dist-test-slave-m28t krb5kdc[17797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612551, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:31.581947 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:31.571242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:36103 (local address 127.13.88.190:44507)
0128 15:02:31.571378 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:31.571382 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:31.572067 (+ 685us) server_negotiation.cc:408] Connection header received
0128 15:02:31.572977 (+ 910us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:31.572980 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:31.573028 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:31.573101 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:31.574764 (+ 1663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.575369 (+ 605us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:31.576116 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.576326 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:31.578549 (+ 2223us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:31.578566 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:31.578568 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:31.578594 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:31.580035 (+ 1441us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:31.580533 (+ 498us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:31.580537 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:31.580539 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:31.580598 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:31.580869 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:31.580872 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:31.580874 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:31.581078 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:31.581176 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:31.581641 (+ 465us) server_negotiation.cc:300] Negotiation successful
0128 15:02:31.581767 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260128 15:02:31.582440 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:31.571502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:44507 (local address 127.13.88.130:36103)
0128 15:02:31.571907 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:31.571941 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:31.572789 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:31.573241 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:31.573248 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:31.573598 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:31.574580 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:31.574595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.575499 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:31.575503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:31.576003 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:31.576012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.576232 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:31.576831 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:31.576856 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:31.578384 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:31.580148 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:31.580152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:31.580161 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:31.580380 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:31.580685 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:31.580687 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:31.580688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:31.580772 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:31.581171 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:31.581176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:31.581391 (+ 215us) client_negotiation.cc:769] Sending connection context
0128 15:02:31.581550 (+ 159us) client_negotiation.cc:240] Negotiation successful
0128 15:02:31.581755 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.13.88.131@KRBTEST.COM; defaulting to no policy
I20260128 15:02:31.583499 18174 heartbeater.cc:344] Connected to a master server at 127.13.88.190:44507
I20260128 15:02:31.583751 18174 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:31.584254 18174 heartbeater.cc:507] Master 127.13.88.190:44507 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.131@KRBTEST.COM" created.
I20260128 15:02:31.585363 17845 ts_manager.cc:194] Registered new tserver with Master: d835d9d5f725430293f991eca1915290 (127.13.88.130:38493)
I20260128 15:02:31.586043 17845 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.130@KRBTEST.COM'} at 127.13.88.130:36103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:31.617318 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:44507
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:45653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:31.721544 18181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:31.721768 18181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:31.721836 18181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:31.725096 18181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:31.725155 18181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:31.725255 18181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:02:31.729516 18181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:45653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:44507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:31.730513 18181 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:31.731310 18181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:31.737802 18186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:31.737841 18187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:31.737811 18189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:31.738116 18181 server_base.cc:1060] running on GCE node
I20260128 15:02:31.738451 18181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:31.738960 18181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:31.740109 18181 hybrid_clock.cc:648] HybridClock initialized: now 1769612551740096 us; error 25 us; skew 500 ppm
Jan 28 15:02:31 dist-test-slave-m28t krb5kdc[17797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612551, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:31.742517 18181 init.cc:377] Logged in from keytab as kudu/127.13.88.131@KRBTEST.COM (short username kudu)
I20260128 15:02:31.743460 18181 webserver.cc:492] Webserver started at http://127.13.88.131:33517/ using document root <none> and password file <none>
I20260128 15:02:31.743956 18181 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:31.744011 18181 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:31.744179 18181 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:31.745826 18181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "e580a1c3c2a24b089786d9797a151a7f"
format_stamp: "Formatted at 2026-01-28 15:02:31 on dist-test-slave-m28t"
server_key: "0c91d81436779684bbbece60ff160c4c"
server_key_iv: "b477e47def882297dcb134cc42dc794f"
server_key_version: "encryptionkey@0"
I20260128 15:02:31.746277 18181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "e580a1c3c2a24b089786d9797a151a7f"
format_stamp: "Formatted at 2026-01-28 15:02:31 on dist-test-slave-m28t"
server_key: "0c91d81436779684bbbece60ff160c4c"
server_key_iv: "b477e47def882297dcb134cc42dc794f"
server_key_version: "encryptionkey@0"
I20260128 15:02:31.749518 18181 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260128 15:02:31.751523 18196 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:31.752471 18181 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260128 15:02:31.752565 18181 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "e580a1c3c2a24b089786d9797a151a7f"
format_stamp: "Formatted at 2026-01-28 15:02:31 on dist-test-slave-m28t"
server_key: "0c91d81436779684bbbece60ff160c4c"
server_key_iv: "b477e47def882297dcb134cc42dc794f"
server_key_version: "encryptionkey@0"
I20260128 15:02:31.752641 18181 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:31.769716 18181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:31.772522 18181 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:31.772683 18181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:31.773207 18181 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:31.774080 18181 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:31.774134 18181 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:31.774187 18181 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:31.774211 18181 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:31.782486 18181 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:37493
I20260128 15:02:31.782517 18309 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:37493 every 8 connection(s)
I20260128 15:02:31.783390 18181 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
Jan 28 15:02:31 dist-test-slave-m28t krb5kdc[17797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612551, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:31.791972 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 18181
I20260128 15:02:31.792094 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:02:31.792368 13666 external_mini_cluster.cc:1468] Setting key 26bbf23e1c5dbcae9194e44ad53c2666
I20260128 15:02:31.795584 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:31.785059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:33705 (local address 127.13.88.190:44507)
0128 15:02:31.785195 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:31.785199 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:31.785907 (+ 708us) server_negotiation.cc:408] Connection header received
0128 15:02:31.786761 (+ 854us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:31.786766 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:31.786830 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:31.786902 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:31.788580 (+ 1678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.789085 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:31.789764 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.789942 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:31.791965 (+ 2023us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:31.791987 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:31.791989 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:31.792018 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:31.793681 (+ 1663us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:31.794258 (+ 577us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:31.794261 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:31.794262 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:31.794312 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:31.794631 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:31.794633 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:31.794635 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:31.794800 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:31.794874 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:31.795365 (+ 491us) server_negotiation.cc:300] Negotiation successful
0128 15:02:31.795466 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260128 15:02:31.796200 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:31.785287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:44507 (local address 127.13.88.131:33705)
0128 15:02:31.785748 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:31.785785 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:31.786551 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:31.787081 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:31.787090 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:31.787473 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:31.788392 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:31.788406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.789203 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:31.789207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:31.789633 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:31.789640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.789869 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:31.790402 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:31.790427 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:31.791814 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:31.793805 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:31.793811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:31.793823 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:31.794145 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:31.794399 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:31.794403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:31.794405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:31.794530 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:31.794879 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:31.794884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:31.795118 (+ 234us) client_negotiation.cc:769] Sending connection context
0128 15:02:31.795288 (+ 170us) client_negotiation.cc:240] Negotiation successful
0128 15:02:31.795499 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":118,"threads_started":1}
I20260128 15:02:31.797168 18310 heartbeater.cc:344] Connected to a master server at 127.13.88.190:44507
I20260128 15:02:31.797411 18310 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:31.797930 18310 heartbeater.cc:507] Master 127.13.88.190:44507 requested a full tablet report, sending...
I20260128 15:02:31.798882 17845 ts_manager.cc:194] Registered new tserver with Master: e580a1c3c2a24b089786d9797a151a7f (127.13.88.131:37493)
I20260128 15:02:31.799422 17845 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.131@KRBTEST.COM'} at 127.13.88.131:33705
I20260128 15:02:31.806169 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 28 15:02:31 dist-test-slave-m28t krb5kdc[17797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 28 15:02:31 dist-test-slave-m28t krb5kdc[17797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.129@KRBTEST.COM
I20260128 15:02:31.827708 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:31.820280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.13.88.129:34867)
0128 15:02:31.820619 (+ 339us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:31.820627 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:31.820655 (+ 28us) server_negotiation.cc:408] Connection header received
0128 15:02:31.820717 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:31.820721 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:31.820887 (+ 166us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:31.821002 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:31.821718 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.822404 (+ 686us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:31.822975 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.823133 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:31.824985 (+ 1852us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:31.825006 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:31.825017 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:31.825042 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:31.826448 (+ 1406us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:31.826783 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:31.826788 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:31.826790 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:31.826837 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:31.826984 (+ 147us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:31.826986 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:31.826987 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:31.827149 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:31.827299 (+ 150us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:31.827402 (+ 103us) server_negotiation.cc:300] Negotiation successful
0128 15:02:31.827530 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":133,"threads_started":1}
Jan 28 15:02:31 dist-test-slave-m28t krb5kdc[17797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:31.836591 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:31.830193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55428 (local address 127.13.88.190:44507)
0128 15:02:31.830326 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:31.830330 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:31.830344 (+ 14us) server_negotiation.cc:408] Connection header received
0128 15:02:31.830433 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:31.830436 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:31.830480 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:31.830543 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:31.831165 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.831639 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:31.832236 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:31.832463 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:31.834171 (+ 1708us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:31.834187 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:31.834189 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:31.834213 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:31.835258 (+ 1045us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:31.835579 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:31.835582 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:31.835584 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:31.835626 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:31.835838 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:31.835842 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:31.835844 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:31.836012 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:31.836127 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:31.836322 (+ 195us) server_negotiation.cc:300] Negotiation successful
0128 15:02:31.836453 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260128 15:02:31.836966 17845 server_base.cc:1142] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55428
I20260128 15:02:31.837517 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 17909
I20260128 15:02:31.842356 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 18045
I20260128 15:02:31.847107 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 18181
I20260128 15:02:31.851511 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 17813
2026-01-28T15:02:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2925 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:02:31 dist-test-slave-m28t krb5kdc[18322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:02:31 dist-test-slave-m28t krb5kdc[18322](info): set up 2 sockets
Jan 28 15:02:31 dist-test-slave-m28t krb5kdc[18322](info): commencing operation
krb5kdc: starting...
W20260128 15:02:33.692605 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.821s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Jan 28 15:02:33 dist-test-slave-m28t krb5kdc[18322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:02:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:33.816222 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:39261
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:39261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:33.922267 18338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:33.922487 18338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:33.922560 18338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:33.926151 18338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:33.926226 18338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:33.926261 18338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:33.926286 18338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:33.926314 18338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:02:33.930577 18338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:39261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:39261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:33.931677 18338 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:33.932576 18338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:33.937935 18346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:33.938068 18338 server_base.cc:1060] running on GCE node
W20260128 15:02:33.938028 18343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:33.937990 18344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:33.938649 18338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:33.939514 18338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:33.940651 18338 hybrid_clock.cc:648] HybridClock initialized: now 1769612553940640 us; error 23 us; skew 500 ppm
Jan 28 15:02:33 dist-test-slave-m28t krb5kdc[18322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612553, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:33.943002 18338 init.cc:377] Logged in from keytab as kudu/127.13.88.190@KRBTEST.COM (short username kudu)
I20260128 15:02:33.943674 18338 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:33.943735 18338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:33.943913 18338 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:33.945596 18338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "3a9432ca14ec425f84fbd83bb1b895a7"
format_stamp: "Formatted at 2026-01-28 15:02:33 on dist-test-slave-m28t"
server_key: "03bce20ec52e679f36e98928820709c3"
server_key_iv: "2fdf504555b8f4c1f184e6a5f634fb72"
server_key_version: "encryptionkey@0"
I20260128 15:02:33.946053 18338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "3a9432ca14ec425f84fbd83bb1b895a7"
format_stamp: "Formatted at 2026-01-28 15:02:33 on dist-test-slave-m28t"
server_key: "03bce20ec52e679f36e98928820709c3"
server_key_iv: "2fdf504555b8f4c1f184e6a5f634fb72"
server_key_version: "encryptionkey@0"
I20260128 15:02:33.949411 18338 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260128 15:02:33.951560 18352 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:33.952558 18338 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260128 15:02:33.952662 18338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "3a9432ca14ec425f84fbd83bb1b895a7"
format_stamp: "Formatted at 2026-01-28 15:02:33 on dist-test-slave-m28t"
server_key: "03bce20ec52e679f36e98928820709c3"
server_key_iv: "2fdf504555b8f4c1f184e6a5f634fb72"
server_key_version: "encryptionkey@0"
I20260128 15:02:33.952754 18338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:33.982749 18338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:33.985430 18338 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:33.985605 18338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:33.992481 18338 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:39261
I20260128 15:02:33.992478 18404 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:39261 every 8 connection(s)
I20260128 15:02:33.993089 18338 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:33.995721 18405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:34.000941 18405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7: Bootstrap starting.
I20260128 15:02:34.001930 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 18338
I20260128 15:02:34.002036 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:34.002236 13666 external_mini_cluster.cc:1468] Setting key 2996c824ef044db51cc3a302a82d23e9
I20260128 15:02:34.003355 18405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:34.004043 18405 log.cc:826] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:34.006083 18405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7: No bootstrap required, opened a new log
I20260128 15:02:34.009086 18405 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9432ca14ec425f84fbd83bb1b895a7" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39261 } }
I20260128 15:02:34.009274 18405 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:34.009331 18405 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9432ca14ec425f84fbd83bb1b895a7, State: Initialized, Role: FOLLOWER
Jan 28 15:02:34 dist-test-slave-m28t krb5kdc[18322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:34.009765 18405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9432ca14ec425f84fbd83bb1b895a7" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39261 } }
I20260128 15:02:34.009891 18405 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:34.009946 18405 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:34.010041 18405 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:34.010802 18405 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9432ca14ec425f84fbd83bb1b895a7" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39261 } }
I20260128 15:02:34.011123 18405 leader_election.cc:304] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a9432ca14ec425f84fbd83bb1b895a7; no voters:
I20260128 15:02:34.011387 18405 leader_election.cc:290] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:34.011495 18410 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:34.011691 18410 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [term 1 LEADER]: Becoming Leader. State: Replica: 3a9432ca14ec425f84fbd83bb1b895a7, State: Running, Role: LEADER
I20260128 15:02:34.012024 18410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9432ca14ec425f84fbd83bb1b895a7" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39261 } }
I20260128 15:02:34.012545 18405 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:34.013515 18412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a9432ca14ec425f84fbd83bb1b895a7. Latest consensus state: current_term: 1 leader_uuid: "3a9432ca14ec425f84fbd83bb1b895a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9432ca14ec425f84fbd83bb1b895a7" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39261 } } }
I20260128 15:02:34.013622 18412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:34.013907 18411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a9432ca14ec425f84fbd83bb1b895a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9432ca14ec425f84fbd83bb1b895a7" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39261 } } }
I20260128 15:02:34.013975 18411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7 [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:34.014297 18419 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:34.015887 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:34.003181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51870 (local address 127.13.88.190:39261)
0128 15:02:34.003536 (+ 355us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:34.003545 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:34.003578 (+ 33us) server_negotiation.cc:408] Connection header received
0128 15:02:34.004136 (+ 558us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:34.004155 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:34.004507 (+ 352us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:34.004858 (+ 351us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:34.005814 (+ 956us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.006559 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:34.007419 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.007718 (+ 299us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:34.010046 (+ 2328us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:34.010070 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:34.010081 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:34.010124 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:34.012487 (+ 2363us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:34.012932 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:34.012937 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:34.012942 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:34.013021 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:34.013283 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:34.013286 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:34.013288 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:34.013723 (+ 435us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:34.013904 (+ 181us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:34.014846 (+ 942us) server_negotiation.cc:300] Negotiation successful
0128 15:02:34.015118 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":97,"threads_started":1}
I20260128 15:02:34.017318 18419 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:34.022367 18419 catalog_manager.cc:1357] Generated new cluster ID: b5469afbc284497ea53952e753aae487
I20260128 15:02:34.022428 18419 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:34.055739 18419 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:34.056774 18419 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:34.065059 18419 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7: Generated new TSK 0
I20260128 15:02:34.065766 18419 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:34.121030 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:39261
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:34.222506 18433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:34.222725 18433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:34.222783 18433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:34.225977 18433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:34.226034 18433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:34.226117 18433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:02:34.230185 18433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:39261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:34.231137 18433 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:34.231931 18433 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:34.237805 18438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:34.237814 18441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:34.237816 18439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:34.238086 18433 server_base.cc:1060] running on GCE node
I20260128 15:02:34.238440 18433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:34.238936 18433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:34.240084 18433 hybrid_clock.cc:648] HybridClock initialized: now 1769612554240073 us; error 26 us; skew 500 ppm
Jan 28 15:02:34 dist-test-slave-m28t krb5kdc[18322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612554, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:34.242455 18433 init.cc:377] Logged in from keytab as kudu/127.13.88.129@KRBTEST.COM (short username kudu)
I20260128 15:02:34.243134 18433 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:34.243201 18433 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:34.243378 18433 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:34.245057 18433 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "fb8107683ef444fea3540f16fae60ab4"
format_stamp: "Formatted at 2026-01-28 15:02:34 on dist-test-slave-m28t"
server_key: "755c7b1f8e88f9ee88329955216dcdfb"
server_key_iv: "804499e1790377f0bbb129f7a2591a0d"
server_key_version: "encryptionkey@0"
I20260128 15:02:34.245486 18433 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "fb8107683ef444fea3540f16fae60ab4"
format_stamp: "Formatted at 2026-01-28 15:02:34 on dist-test-slave-m28t"
server_key: "755c7b1f8e88f9ee88329955216dcdfb"
server_key_iv: "804499e1790377f0bbb129f7a2591a0d"
server_key_version: "encryptionkey@0"
I20260128 15:02:34.248667 18433 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260128 15:02:34.250756 18447 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:34.251695 18433 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:34.251793 18433 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "fb8107683ef444fea3540f16fae60ab4"
format_stamp: "Formatted at 2026-01-28 15:02:34 on dist-test-slave-m28t"
server_key: "755c7b1f8e88f9ee88329955216dcdfb"
server_key_iv: "804499e1790377f0bbb129f7a2591a0d"
server_key_version: "encryptionkey@0"
I20260128 15:02:34.251888 18433 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:34.265043 18433 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:34.267350 18433 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:34.267508 18433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:34.268003 18433 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:34.268872 18433 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:34.268918 18433 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:34.268966 18433 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:34.268994 18433 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:34.277230 18433 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:35563
I20260128 15:02:34.277257 18560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:35563 every 8 connection(s)
I20260128 15:02:34.277730 18433 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
Jan 28 15:02:34 dist-test-slave-m28t krb5kdc[18322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612554, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:34.285777 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 18433
I20260128 15:02:34.285895 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:02:34.286167 13666 external_mini_cluster.cc:1468] Setting key 5f765135a4a2d3c4a218b37f0b47e7d1
I20260128 15:02:34.289634 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:34.279319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:59101 (local address 127.13.88.190:39261)
0128 15:02:34.279439 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:34.279443 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:34.280064 (+ 621us) server_negotiation.cc:408] Connection header received
0128 15:02:34.280926 (+ 862us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:34.280930 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:34.280997 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:34.281072 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:34.282649 (+ 1577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.283173 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:34.283811 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.283995 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:34.286131 (+ 2136us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:34.286152 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:34.286154 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:34.286183 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:34.287565 (+ 1382us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:34.288127 (+ 562us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:34.288130 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:34.288132 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:34.288174 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:34.288460 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:34.288463 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:34.288464 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:34.288615 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:34.288686 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:34.289322 (+ 636us) server_negotiation.cc:300] Negotiation successful
0128 15:02:34.289472 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260128 15:02:34.290208 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:34.279523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39261 (local address 127.13.88.129:59101)
0128 15:02:34.279922 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:34.279957 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:34.280727 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:34.281236 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:34.281244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:34.281607 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:34.282497 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:34.282510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.283286 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:34.283289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:34.283693 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:34.283700 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.283882 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:34.284447 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:34.284472 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:34.285942 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:34.287693 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:34.287699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:34.287711 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:34.288044 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:34.288242 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:34.288245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:34.288247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:34.288364 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:34.288699 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:34.288706 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:34.289022 (+ 316us) client_negotiation.cc:769] Sending connection context
0128 15:02:34.289260 (+ 238us) client_negotiation.cc:240] Negotiation successful
0128 15:02:34.289492 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":101,"threads_started":1}
I20260128 15:02:34.291493 18561 heartbeater.cc:344] Connected to a master server at 127.13.88.190:39261
I20260128 15:02:34.291730 18561 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:34.292210 18561 heartbeater.cc:507] Master 127.13.88.190:39261 requested a full tablet report, sending...
I20260128 15:02:34.293614 18369 ts_manager.cc:194] Registered new tserver with Master: fb8107683ef444fea3540f16fae60ab4 (127.13.88.129:35563)
WARNING: no policy specified for kudu/127.13.88.130@KRBTEST.COM; defaulting to no policy
I20260128 15:02:34.294771 18369 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.129@KRBTEST.COM'} at 127.13.88.129:59101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:34.327312 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:39261
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:34.429100 18568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:34.429330 18568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:34.429406 18568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:34.432610 18568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:34.432665 18568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:34.432754 18568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:02:34.436883 18568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:39261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:34.437930 18568 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:34.438759 18568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:34.445011 18576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:34.445011 18574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:34.445011 18573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:34.445201 18568 server_base.cc:1060] running on GCE node
I20260128 15:02:34.445622 18568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:34.446126 18568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:34.447263 18568 hybrid_clock.cc:648] HybridClock initialized: now 1769612554447246 us; error 27 us; skew 500 ppm
Jan 28 15:02:34 dist-test-slave-m28t krb5kdc[18322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612554, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:34.449685 18568 init.cc:377] Logged in from keytab as kudu/127.13.88.130@KRBTEST.COM (short username kudu)
I20260128 15:02:34.450371 18568 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:34.450431 18568 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:34.450598 18568 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:34.452270 18568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "3cad8605f4914e5a8f2e7f592c3f9a73"
format_stamp: "Formatted at 2026-01-28 15:02:34 on dist-test-slave-m28t"
server_key: "ede922fce0b948e8c717949d693e30c7"
server_key_iv: "fc104a0b8fb8a67bb2b92178c175c331"
server_key_version: "encryptionkey@0"
I20260128 15:02:34.452744 18568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "3cad8605f4914e5a8f2e7f592c3f9a73"
format_stamp: "Formatted at 2026-01-28 15:02:34 on dist-test-slave-m28t"
server_key: "ede922fce0b948e8c717949d693e30c7"
server_key_iv: "fc104a0b8fb8a67bb2b92178c175c331"
server_key_version: "encryptionkey@0"
I20260128 15:02:34.455850 18568 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260128 15:02:34.457909 18582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:34.458835 18568 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260128 15:02:34.458925 18568 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "3cad8605f4914e5a8f2e7f592c3f9a73"
format_stamp: "Formatted at 2026-01-28 15:02:34 on dist-test-slave-m28t"
server_key: "ede922fce0b948e8c717949d693e30c7"
server_key_iv: "fc104a0b8fb8a67bb2b92178c175c331"
server_key_version: "encryptionkey@0"
I20260128 15:02:34.459013 18568 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:34.469844 18568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:34.472242 18568 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:34.472394 18568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:34.472915 18568 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:34.473745 18568 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:34.473790 18568 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:34.473831 18568 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:34.473860 18568 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:34.482203 18568 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:38089
I20260128 15:02:34.482239 18695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:38089 every 8 connection(s)
I20260128 15:02:34.482687 18568 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
Jan 28 15:02:34 dist-test-slave-m28t krb5kdc[18322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612554, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:34.492261 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 18568
I20260128 15:02:34.492388 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:02:34.492645 13666 external_mini_cluster.cc:1468] Setting key c7c308d6ca9362c2ed3dbeb743141aed
I20260128 15:02:34.493777 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:34.484156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:38795 (local address 127.13.88.190:39261)
0128 15:02:34.484288 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:34.484292 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:34.484943 (+ 651us) server_negotiation.cc:408] Connection header received
0128 15:02:34.485755 (+ 812us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:34.485759 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:34.485817 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:34.485889 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:34.487449 (+ 1560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.487940 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:34.488595 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.488773 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:34.490731 (+ 1958us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:34.490751 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:34.490753 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:34.490778 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:34.491843 (+ 1065us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:34.492315 (+ 472us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:34.492320 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:34.492322 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:34.492381 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:34.492690 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:34.492694 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:34.492695 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:34.492888 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:34.492987 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:34.493507 (+ 520us) server_negotiation.cc:300] Negotiation successful
0128 15:02:34.493622 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260128 15:02:34.494354 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:34.484390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39261 (local address 127.13.88.130:38795)
0128 15:02:34.484816 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:34.484851 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:34.485555 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:34.486050 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:34.486058 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:34.486429 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:34.487305 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:34.487318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.488052 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:34.488055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:34.488490 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:34.488498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.488679 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:34.489230 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:34.489253 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:34.490600 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:34.491973 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:34.491978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:34.491988 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:34.492212 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:34.492497 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:34.492500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:34.492502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:34.492598 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:34.493022 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:34.493029 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:34.493288 (+ 259us) client_negotiation.cc:769] Sending connection context
0128 15:02:34.493457 (+ 169us) client_negotiation.cc:240] Negotiation successful
0128 15:02:34.493660 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":96,"threads_started":1}
I20260128 15:02:34.495422 18696 heartbeater.cc:344] Connected to a master server at 127.13.88.190:39261
I20260128 15:02:34.495638 18696 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:34.496090 18696 heartbeater.cc:507] Master 127.13.88.190:39261 requested a full tablet report, sending...
I20260128 15:02:34.497058 18369 ts_manager.cc:194] Registered new tserver with Master: 3cad8605f4914e5a8f2e7f592c3f9a73 (127.13.88.130:38089)
I20260128 15:02:34.497772 18369 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.130@KRBTEST.COM'} at 127.13.88.130:38795
WARNING: no policy specified for kudu/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:34.531340 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:39261
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:34.634306 18703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:34.634511 18703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:34.634568 18703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:34.637801 18703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:34.637856 18703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:34.637934 18703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:02:34.641929 18703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:39261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:34.642871 18703 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:34.643620 18703 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:34.649513 18709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:34.649519 18708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:34.649536 18711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:34.649699 18703 server_base.cc:1060] running on GCE node
I20260128 15:02:34.650110 18703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:34.650590 18703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:34.651726 18703 hybrid_clock.cc:648] HybridClock initialized: now 1769612554651705 us; error 38 us; skew 500 ppm
Jan 28 15:02:34 dist-test-slave-m28t krb5kdc[18322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612554, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:34.654062 18703 init.cc:377] Logged in from keytab as kudu/127.13.88.131@KRBTEST.COM (short username kudu)
I20260128 15:02:34.654706 18703 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:34.654767 18703 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:34.654912 18703 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:34.656603 18703 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "6355db46001a45bfb188dbeeb571c028"
format_stamp: "Formatted at 2026-01-28 15:02:34 on dist-test-slave-m28t"
server_key: "a734c04b3744eeffd18e487d5d33c2d6"
server_key_iv: "bf8d86f9f33abfeef6fc46741baacd45"
server_key_version: "encryptionkey@0"
I20260128 15:02:34.657013 18703 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "6355db46001a45bfb188dbeeb571c028"
format_stamp: "Formatted at 2026-01-28 15:02:34 on dist-test-slave-m28t"
server_key: "a734c04b3744eeffd18e487d5d33c2d6"
server_key_iv: "bf8d86f9f33abfeef6fc46741baacd45"
server_key_version: "encryptionkey@0"
I20260128 15:02:34.659988 18703 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260128 15:02:34.661942 18717 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:34.662858 18703 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:02:34.662941 18703 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "6355db46001a45bfb188dbeeb571c028"
format_stamp: "Formatted at 2026-01-28 15:02:34 on dist-test-slave-m28t"
server_key: "a734c04b3744eeffd18e487d5d33c2d6"
server_key_iv: "bf8d86f9f33abfeef6fc46741baacd45"
server_key_version: "encryptionkey@0"
I20260128 15:02:34.663010 18703 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:34.678130 18703 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:34.680969 18703 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:34.681118 18703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:34.681607 18703 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:34.682433 18703 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:34.682482 18703 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:34.682518 18703 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:34.682538 18703 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:34.691496 18703 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:39565
I20260128 15:02:34.691524 18830 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:39565 every 8 connection(s)
I20260128 15:02:34.692016 18703 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
I20260128 15:02:34.696318 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 18703
I20260128 15:02:34.696456 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:02:34.696723 13666 external_mini_cluster.cc:1468] Setting key 8d1eea611d6ec4d5fba462577719e8fc
Jan 28 15:02:34 dist-test-slave-m28t krb5kdc[18322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612554, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:34.703158 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:34.693581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:48939 (local address 127.13.88.190:39261)
0128 15:02:34.693703 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:34.693707 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:34.694363 (+ 656us) server_negotiation.cc:408] Connection header received
0128 15:02:34.695128 (+ 765us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:34.695130 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:34.695181 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:34.695251 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:34.697089 (+ 1838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.697730 (+ 641us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:34.698434 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.698616 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:34.700473 (+ 1857us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:34.700488 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:34.700490 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:34.700515 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:34.701767 (+ 1252us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:34.702131 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:34.702134 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:34.702135 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:34.702174 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:34.702360 (+ 186us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:34.702363 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:34.702364 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:34.702492 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:34.702558 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:34.702951 (+ 393us) server_negotiation.cc:300] Negotiation successful
0128 15:02:34.703046 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260128 15:02:34.703742 18833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:34.693823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39261 (local address 127.13.88.131:48939)
0128 15:02:34.694226 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:34.694260 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:34.694954 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:34.695383 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:34.695390 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:34.695753 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:34.696910 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:34.696929 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.697867 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:34.697871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:34.698322 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:34.698330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.698508 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:34.698987 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:34.699010 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:34.700323 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:34.701851 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:34.701855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:34.701862 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:34.702060 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:34.702232 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:34.702234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:34.702236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:34.702305 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:34.702555 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:34.702560 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:34.702757 (+ 197us) client_negotiation.cc:769] Sending connection context
0128 15:02:34.702920 (+ 163us) client_negotiation.cc:240] Negotiation successful
0128 15:02:34.703107 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
I20260128 15:02:34.704706 18831 heartbeater.cc:344] Connected to a master server at 127.13.88.190:39261
I20260128 15:02:34.704921 18831 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:34.705338 18831 heartbeater.cc:507] Master 127.13.88.190:39261 requested a full tablet report, sending...
I20260128 15:02:34.706197 18369 ts_manager.cc:194] Registered new tserver with Master: 6355db46001a45bfb188dbeeb571c028 (127.13.88.131:39565)
I20260128 15:02:34.706713 18369 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.131@KRBTEST.COM'} at 127.13.88.131:48939
I20260128 15:02:34.709890 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260128 15:02:34.717353 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:34.711569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.13.88.190:39261)
0128 15:02:34.711691 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:34.711696 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:34.711826 (+ 130us) server_negotiation.cc:408] Connection header received
0128 15:02:34.711912 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:34.711915 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:34.711965 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:34.712028 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:34.712768 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.713236 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:34.713776 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.713911 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:34.714880 (+ 969us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:34.714894 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:34.714896 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:34.714917 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:34.716134 (+ 1217us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:34.716507 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:34.716510 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:34.716512 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:34.716557 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:34.716732 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:34.716734 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:34.716735 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:34.716880 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:34.716981 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:34.717118 (+ 137us) server_negotiation.cc:300] Negotiation successful
0128 15:02:34.717210 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260128 15:02:34.719527 18369 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 15:02:34.721964 18369 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:02:34.733918 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:34.729879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:35563 (local address 127.0.0.1:54806)
0128 15:02:34.730660 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:34.730671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:34.730753 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:34.731179 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:34.731181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:34.731198 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:34.731442 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:34.731449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.732754 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:34.732758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:34.733487 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:34.733496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.733587 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:34.733635 (+ 48us) client_negotiation.cc:769] Sending connection context
0128 15:02:34.733715 (+ 80us) client_negotiation.cc:240] Negotiation successful
0128 15:02:34.733790 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":79,"threads_started":1}
I20260128 15:02:34.733953 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:34.729362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:38089 (local address 127.0.0.1:36072)
0128 15:02:34.729955 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:34.729992 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:34.730133 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:34.730839 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:34.730844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:34.730866 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:34.731156 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:34.731164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.732562 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:34.732565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:34.733476 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:34.733486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.733582 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:34.733634 (+ 52us) client_negotiation.cc:769] Sending connection context
0128 15:02:34.733711 (+ 77us) client_negotiation.cc:240] Negotiation successful
0128 15:02:34.733795 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":73,"threads_started":1}
I20260128 15:02:34.734381 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:34.729574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:39565 (local address 127.0.0.1:57540)
0128 15:02:34.730104 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:34.730118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:34.730202 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:34.731363 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:34.731366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:34.731382 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:34.731609 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:34.731616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.732512 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:34.732516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:34.733561 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:34.733568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.733646 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:34.733658 (+ 12us) client_negotiation.cc:769] Sending connection context
0128 15:02:34.733711 (+ 53us) client_negotiation.cc:240] Negotiation successful
0128 15:02:34.733797 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":96,"threads_started":1}
I20260128 15:02:34.734977 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:34.730236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.13.88.129:35563)
0128 15:02:34.730546 (+ 310us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:34.730552 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:34.730843 (+ 291us) server_negotiation.cc:408] Connection header received
0128 15:02:34.730924 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:34.730929 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:34.731062 (+ 133us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:34.731178 (+ 116us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:34.731562 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.732653 (+ 1091us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:34.734113 (+ 1460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.734656 (+ 543us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:34.734756 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:34.734805 (+ 49us) server_negotiation.cc:300] Negotiation successful
0128 15:02:34.734871 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":105,"threads_started":1}
I20260128 15:02:34.735041 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:34.729528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36072 (local address 127.13.88.130:38089)
0128 15:02:34.730387 (+ 859us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:34.730394 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:34.730416 (+ 22us) server_negotiation.cc:408] Connection header received
0128 15:02:34.730494 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:34.730499 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:34.730671 (+ 172us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:34.730839 (+ 168us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:34.731292 (+ 453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.732437 (+ 1145us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:34.733879 (+ 1442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.734688 (+ 809us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:34.734779 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:34.734828 (+ 49us) server_negotiation.cc:300] Negotiation successful
0128 15:02:34.734902 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":103,"threads_started":1}
I20260128 15:02:34.735033 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:34.730058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.13.88.131:39565)
0128 15:02:34.730984 (+ 926us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:34.730990 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:34.731012 (+ 22us) server_negotiation.cc:408] Connection header received
0128 15:02:34.731084 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:34.731089 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:34.731240 (+ 151us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:34.731370 (+ 130us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:34.731704 (+ 334us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.732392 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:34.733916 (+ 1524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:34.734649 (+ 733us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:34.734758 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:34.734829 (+ 71us) server_negotiation.cc:300] Negotiation successful
0128 15:02:34.734905 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":153,"threads_started":1}
I20260128 15:02:34.737627 18495 tablet_service.cc:1505] Processing CreateTablet for tablet 3ed4febaf9554ba6943d4ba7a994e873 (DEFAULT_TABLE table=test-table [id=9a8f94b1947a4e2eb171cff4f53ad5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:34.737679 18630 tablet_service.cc:1505] Processing CreateTablet for tablet 3ed4febaf9554ba6943d4ba7a994e873 (DEFAULT_TABLE table=test-table [id=9a8f94b1947a4e2eb171cff4f53ad5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:34.737679 18765 tablet_service.cc:1505] Processing CreateTablet for tablet 3ed4febaf9554ba6943d4ba7a994e873 (DEFAULT_TABLE table=test-table [id=9a8f94b1947a4e2eb171cff4f53ad5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:34.738509 18495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ed4febaf9554ba6943d4ba7a994e873. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:34.738535 18630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ed4febaf9554ba6943d4ba7a994e873. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:34.738535 18765 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ed4febaf9554ba6943d4ba7a994e873. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:34.743541 18848 tablet_bootstrap.cc:492] T 3ed4febaf9554ba6943d4ba7a994e873 P fb8107683ef444fea3540f16fae60ab4: Bootstrap starting.
I20260128 15:02:34.744683 18850 tablet_bootstrap.cc:492] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73: Bootstrap starting.
I20260128 15:02:34.746361 18848 tablet_bootstrap.cc:654] T 3ed4febaf9554ba6943d4ba7a994e873 P fb8107683ef444fea3540f16fae60ab4: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:34.746361 18850 tablet_bootstrap.cc:654] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:34.747171 18848 log.cc:826] T 3ed4febaf9554ba6943d4ba7a994e873 P fb8107683ef444fea3540f16fae60ab4: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:34.747171 18850 log.cc:826] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:34.747758 18849 tablet_bootstrap.cc:492] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028: Bootstrap starting.
I20260128 15:02:34.748836 18850 tablet_bootstrap.cc:492] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73: No bootstrap required, opened a new log
I20260128 15:02:34.749004 18850 ts_tablet_manager.cc:1403] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260128 15:02:34.749251 18848 tablet_bootstrap.cc:492] T 3ed4febaf9554ba6943d4ba7a994e873 P fb8107683ef444fea3540f16fae60ab4: No bootstrap required, opened a new log
I20260128 15:02:34.749421 18848 ts_tablet_manager.cc:1403] T 3ed4febaf9554ba6943d4ba7a994e873 P fb8107683ef444fea3540f16fae60ab4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260128 15:02:34.749545 18849 tablet_bootstrap.cc:654] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:34.750272 18849 log.cc:826] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:34.751968 18849 tablet_bootstrap.cc:492] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028: No bootstrap required, opened a new log
I20260128 15:02:34.752141 18849 ts_tablet_manager.cc:1403] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260128 15:02:34.752231 18850 raft_consensus.cc:359] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad8605f4914e5a8f2e7f592c3f9a73" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 38089 } } peers { permanent_uuid: "6355db46001a45bfb188dbeeb571c028" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 39565 } } peers { permanent_uuid: "fb8107683ef444fea3540f16fae60ab4" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35563 } }
I20260128 15:02:34.752456 18850 raft_consensus.cc:385] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:34.752333 18848 raft_consensus.cc:359] T 3ed4febaf9554ba6943d4ba7a994e873 P fb8107683ef444fea3540f16fae60ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad8605f4914e5a8f2e7f592c3f9a73" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 38089 } } peers { permanent_uuid: "6355db46001a45bfb188dbeeb571c028" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 39565 } } peers { permanent_uuid: "fb8107683ef444fea3540f16fae60ab4" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35563 } }
I20260128 15:02:34.752524 18848 raft_consensus.cc:385] T 3ed4febaf9554ba6943d4ba7a994e873 P fb8107683ef444fea3540f16fae60ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:34.752553 18850 raft_consensus.cc:740] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cad8605f4914e5a8f2e7f592c3f9a73, State: Initialized, Role: FOLLOWER
I20260128 15:02:34.752581 18848 raft_consensus.cc:740] T 3ed4febaf9554ba6943d4ba7a994e873 P fb8107683ef444fea3540f16fae60ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb8107683ef444fea3540f16fae60ab4, State: Initialized, Role: FOLLOWER
I20260128 15:02:34.752974 18848 consensus_queue.cc:260] T 3ed4febaf9554ba6943d4ba7a994e873 P fb8107683ef444fea3540f16fae60ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad8605f4914e5a8f2e7f592c3f9a73" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 38089 } } peers { permanent_uuid: "6355db46001a45bfb188dbeeb571c028" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 39565 } } peers { permanent_uuid: "fb8107683ef444fea3540f16fae60ab4" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35563 } }
I20260128 15:02:34.752985 18850 consensus_queue.cc:260] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad8605f4914e5a8f2e7f592c3f9a73" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 38089 } } peers { permanent_uuid: "6355db46001a45bfb188dbeeb571c028" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 39565 } } peers { permanent_uuid: "fb8107683ef444fea3540f16fae60ab4" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35563 } }
I20260128 15:02:34.753517 18561 heartbeater.cc:499] Master 127.13.88.190:39261 was elected leader, sending a full tablet report...
I20260128 15:02:34.753773 18848 ts_tablet_manager.cc:1434] T 3ed4febaf9554ba6943d4ba7a994e873 P fb8107683ef444fea3540f16fae60ab4: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260128 15:02:34.753777 18850 ts_tablet_manager.cc:1434] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260128 15:02:34.754113 18696 heartbeater.cc:499] Master 127.13.88.190:39261 was elected leader, sending a full tablet report...
I20260128 15:02:34.755229 18849 raft_consensus.cc:359] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad8605f4914e5a8f2e7f592c3f9a73" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 38089 } } peers { permanent_uuid: "6355db46001a45bfb188dbeeb571c028" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 39565 } } peers { permanent_uuid: "fb8107683ef444fea3540f16fae60ab4" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35563 } }
I20260128 15:02:34.755378 18849 raft_consensus.cc:385] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:34.755458 18849 raft_consensus.cc:740] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6355db46001a45bfb188dbeeb571c028, State: Initialized, Role: FOLLOWER
I20260128 15:02:34.755800 18849 consensus_queue.cc:260] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad8605f4914e5a8f2e7f592c3f9a73" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 38089 } } peers { permanent_uuid: "6355db46001a45bfb188dbeeb571c028" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 39565 } } peers { permanent_uuid: "fb8107683ef444fea3540f16fae60ab4" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35563 } }
I20260128 15:02:34.756444 18849 ts_tablet_manager.cc:1434] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260128 15:02:34.757128 18831 heartbeater.cc:499] Master 127.13.88.190:39261 was elected leader, sending a full tablet report...
W20260128 15:02:34.778682 18562 tablet.cc:2378] T 3ed4febaf9554ba6943d4ba7a994e873 P fb8107683ef444fea3540f16fae60ab4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260128 15:02:34.942837 18832 tablet.cc:2378] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260128 15:02:34.983577 18697 tablet.cc:2378] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 15:02:35.184823 18855 raft_consensus.cc:493] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:02:35.185017 18855 raft_consensus.cc:515] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad8605f4914e5a8f2e7f592c3f9a73" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 38089 } } peers { permanent_uuid: "6355db46001a45bfb188dbeeb571c028" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 39565 } } peers { permanent_uuid: "fb8107683ef444fea3540f16fae60ab4" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35563 } }
I20260128 15:02:35.186160 18855 leader_election.cc:290] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6355db46001a45bfb188dbeeb571c028 (127.13.88.131:39565), fb8107683ef444fea3540f16fae60ab4 (127.13.88.129:35563)
I20260128 15:02:35.189529 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:35.186363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:39565 (local address 127.13.88.130:57001)
0128 15:02:35.186685 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:35.186699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:35.186834 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:35.187072 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:35.187074 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:35.187090 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:35.187349 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:35.187357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:35.188280 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:35.188283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:35.189125 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:35.189134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:35.189246 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:35.189267 (+ 21us) client_negotiation.cc:769] Sending connection context
0128 15:02:35.189315 (+ 48us) client_negotiation.cc:240] Negotiation successful
0128 15:02:35.189377 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"spinlock_wait_cycles":15872,"thread_start_us":97,"threads_started":1}
I20260128 15:02:35.189548 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:35.186441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:35563 (local address 127.13.88.130:54615)
0128 15:02:35.186685 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:35.186699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:35.186834 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:35.187294 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:35.187298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:35.187312 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:35.187545 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:35.187551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:35.188458 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:35.188461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:35.189252 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:35.189260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:35.189350 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:35.189362 (+ 12us) client_negotiation.cc:769] Sending connection context
0128 15:02:35.189402 (+ 40us) client_negotiation.cc:240] Negotiation successful
0128 15:02:35.189446 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"spinlock_wait_cycles":15232,"thread_start_us":73,"threads_started":1}
I20260128 15:02:35.189951 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:35.186454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:57001 (local address 127.13.88.131:39565)
0128 15:02:35.186579 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:35.186583 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:35.186781 (+ 198us) server_negotiation.cc:408] Connection header received
0128 15:02:35.186930 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:35.186933 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:35.186984 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:35.187050 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:35.187468 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:35.188178 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:35.189232 (+ 1054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:35.189680 (+ 448us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:35.189731 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:35.189781 (+ 50us) server_negotiation.cc:300] Negotiation successful
0128 15:02:35.189837 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260128 15:02:35.190032 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:35.186523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:54615 (local address 127.13.88.129:35563)
0128 15:02:35.186647 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:35.186651 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:35.186811 (+ 160us) server_negotiation.cc:408] Connection header received
0128 15:02:35.187161 (+ 350us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:35.187164 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:35.187206 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:35.187259 (+ 53us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:35.187657 (+ 398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:35.188351 (+ 694us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:35.189356 (+ 1005us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:35.189798 (+ 442us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:35.189828 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:35.189869 (+ 41us) server_negotiation.cc:300] Negotiation successful
0128 15:02:35.189919 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260128 15:02:35.190555 18515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ed4febaf9554ba6943d4ba7a994e873" candidate_uuid: "3cad8605f4914e5a8f2e7f592c3f9a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb8107683ef444fea3540f16fae60ab4" is_pre_election: true
I20260128 15:02:35.190555 18780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ed4febaf9554ba6943d4ba7a994e873" candidate_uuid: "3cad8605f4914e5a8f2e7f592c3f9a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6355db46001a45bfb188dbeeb571c028" is_pre_election: true
I20260128 15:02:35.190829 18515 raft_consensus.cc:2468] T 3ed4febaf9554ba6943d4ba7a994e873 P fb8107683ef444fea3540f16fae60ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cad8605f4914e5a8f2e7f592c3f9a73 in term 0.
I20260128 15:02:35.190829 18780 raft_consensus.cc:2468] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cad8605f4914e5a8f2e7f592c3f9a73 in term 0.
I20260128 15:02:35.191318 18585 leader_election.cc:304] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cad8605f4914e5a8f2e7f592c3f9a73, fb8107683ef444fea3540f16fae60ab4; no voters:
I20260128 15:02:35.191560 18855 raft_consensus.cc:2804] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:02:35.191635 18855 raft_consensus.cc:493] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:02:35.191682 18855 raft_consensus.cc:3060] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:35.192597 18855 raft_consensus.cc:515] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad8605f4914e5a8f2e7f592c3f9a73" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 38089 } } peers { permanent_uuid: "6355db46001a45bfb188dbeeb571c028" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 39565 } } peers { permanent_uuid: "fb8107683ef444fea3540f16fae60ab4" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35563 } }
I20260128 15:02:35.192920 18855 leader_election.cc:290] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [CANDIDATE]: Term 1 election: Requested vote from peers 6355db46001a45bfb188dbeeb571c028 (127.13.88.131:39565), fb8107683ef444fea3540f16fae60ab4 (127.13.88.129:35563)
I20260128 15:02:35.193243 18515 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ed4febaf9554ba6943d4ba7a994e873" candidate_uuid: "3cad8605f4914e5a8f2e7f592c3f9a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb8107683ef444fea3540f16fae60ab4"
I20260128 15:02:35.193246 18780 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ed4febaf9554ba6943d4ba7a994e873" candidate_uuid: "3cad8605f4914e5a8f2e7f592c3f9a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6355db46001a45bfb188dbeeb571c028"
I20260128 15:02:35.193353 18515 raft_consensus.cc:3060] T 3ed4febaf9554ba6943d4ba7a994e873 P fb8107683ef444fea3540f16fae60ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:35.193358 18780 raft_consensus.cc:3060] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:35.194239 18515 raft_consensus.cc:2468] T 3ed4febaf9554ba6943d4ba7a994e873 P fb8107683ef444fea3540f16fae60ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cad8605f4914e5a8f2e7f592c3f9a73 in term 1.
I20260128 15:02:35.194286 18780 raft_consensus.cc:2468] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cad8605f4914e5a8f2e7f592c3f9a73 in term 1.
I20260128 15:02:35.194511 18585 leader_election.cc:304] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cad8605f4914e5a8f2e7f592c3f9a73, fb8107683ef444fea3540f16fae60ab4; no voters:
I20260128 15:02:35.194701 18855 raft_consensus.cc:2804] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:35.194896 18855 raft_consensus.cc:697] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [term 1 LEADER]: Becoming Leader. State: Replica: 3cad8605f4914e5a8f2e7f592c3f9a73, State: Running, Role: LEADER
I20260128 15:02:35.195195 18855 consensus_queue.cc:237] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad8605f4914e5a8f2e7f592c3f9a73" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 38089 } } peers { permanent_uuid: "6355db46001a45bfb188dbeeb571c028" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 39565 } } peers { permanent_uuid: "fb8107683ef444fea3540f16fae60ab4" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35563 } }
I20260128 15:02:35.198355 18369 catalog_manager.cc:5654] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cad8605f4914e5a8f2e7f592c3f9a73 (127.13.88.130). New cstate: current_term: 1 leader_uuid: "3cad8605f4914e5a8f2e7f592c3f9a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad8605f4914e5a8f2e7f592c3f9a73" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 38089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6355db46001a45bfb188dbeeb571c028" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 39565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb8107683ef444fea3540f16fae60ab4" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35563 } health_report { overall_health: UNKNOWN } } }
I20260128 15:02:35.291339 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:35.288093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36080 (local address 127.13.88.130:38089)
0128 15:02:35.288337 (+ 244us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:35.288341 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:35.288354 (+ 13us) server_negotiation.cc:408] Connection header received
0128 15:02:35.288550 (+ 196us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:35.288553 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:35.288612 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:35.288701 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:02:35.289080 (+ 379us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:35.289577 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:35.290220 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:35.290404 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:35.290501 (+ 97us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:02:35.290955 (+ 454us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:02:35.291069 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:35.291147 (+ 78us) server_negotiation.cc:300] Negotiation successful
0128 15:02:35.291202 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":63,"threads_started":1}
I20260128 15:02:35.296558 18780 raft_consensus.cc:1275] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028 [term 1 FOLLOWER]: Refusing update from remote peer 3cad8605f4914e5a8f2e7f592c3f9a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:35.296558 18515 raft_consensus.cc:1275] T 3ed4febaf9554ba6943d4ba7a994e873 P fb8107683ef444fea3540f16fae60ab4 [term 1 FOLLOWER]: Refusing update from remote peer 3cad8605f4914e5a8f2e7f592c3f9a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:35.297178 18859 consensus_queue.cc:1048] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb8107683ef444fea3540f16fae60ab4" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:35.297334 18855 consensus_queue.cc:1048] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6355db46001a45bfb188dbeeb571c028" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:35.307571 18868 mvcc.cc:204] Tried to move back new op lower bound from 7248333026487566336 to 7248333026082062336. Current Snapshot: MvccSnapshot[applied={T|T < 7248333026487566336 or (T in {7248333026487566336})}]
I20260128 15:02:35.311717 18369 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:02:35.311889 18369 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:02:35.314152 18369 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 3a9432ca14ec425f84fbd83bb1b895a7: Sending DeleteTablet for 3 replicas of tablet 3ed4febaf9554ba6943d4ba7a994e873
I20260128 15:02:35.314873 18765 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ed4febaf9554ba6943d4ba7a994e873 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:35 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:57540
I20260128 15:02:35.315024 18630 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ed4febaf9554ba6943d4ba7a994e873 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:35 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:36072
I20260128 15:02:35.315373 18495 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ed4febaf9554ba6943d4ba7a994e873 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:35 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:54806
I20260128 15:02:35.315670 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 18433
I20260128 15:02:35.316272 18877 tablet_replica.cc:333] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73: stopping tablet replica
I20260128 15:02:35.316562 18877 raft_consensus.cc:2243] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [term 1 LEADER]: Raft consensus shutting down.
I20260128 15:02:35.316917 18877 raft_consensus.cc:2272] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:35.317137 18876 tablet_replica.cc:333] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028: stopping tablet replica
I20260128 15:02:35.317376 18876 raft_consensus.cc:2243] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:02:35.317682 18876 raft_consensus.cc:2272] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:35.318981 18876 ts_tablet_manager.cc:1916] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:35.321309 18877 ts_tablet_manager.cc:1916] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:35.321564 18876 ts_tablet_manager.cc:1929] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:02:35.321739 18876 log.cc:1199] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-2/wal/wals/3ed4febaf9554ba6943d4ba7a994e873
I20260128 15:02:35.322118 18876 ts_tablet_manager.cc:1950] T 3ed4febaf9554ba6943d4ba7a994e873 P 6355db46001a45bfb188dbeeb571c028: Deleting consensus metadata
W20260128 15:02:35.322425 18355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.88.129:35563 (error 108)
I20260128 15:02:35.323693 18354 catalog_manager.cc:4985] TS 6355db46001a45bfb188dbeeb571c028 (127.13.88.131:39565): tablet 3ed4febaf9554ba6943d4ba7a994e873 (table test-table [id=9a8f94b1947a4e2eb171cff4f53ad5f8]) successfully deleted
I20260128 15:02:35.323768 18877 ts_tablet_manager.cc:1929] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:02:35.323805 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 18568
I20260128 15:02:35.323848 18877 log.cc:1199] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1769612519049359-13666-0/minicluster-data/ts-1/wal/wals/3ed4febaf9554ba6943d4ba7a994e873
I20260128 15:02:35.324121 18877 ts_tablet_manager.cc:1950] T 3ed4febaf9554ba6943d4ba7a994e873 P 3cad8605f4914e5a8f2e7f592c3f9a73: Deleting consensus metadata
I20260128 15:02:35.324934 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:35.323502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:35563 (local address 127.0.0.1:54814)
0128 15:02:35.324779 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:35.324842 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.129:35563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1196,"thread_start_us":67,"threads_started":1}
W20260128 15:02:35.325062 18355 catalog_manager.cc:4712] TS fb8107683ef444fea3540f16fae60ab4 (127.13.88.129:35563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ed4febaf9554ba6943d4ba7a994e873: Network error: Client connection negotiation failed: client connection to 127.13.88.129:35563: connect: Connection refused (error 111)
I20260128 15:02:35.329085 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 18703
I20260128 15:02:35.329497 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:35.329237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:38089 (local address 127.0.0.1:36090)
0128 15:02:35.329389 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:35.329426 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.130:38089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260128 15:02:35.333461 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 18338
2026-01-28T15:02:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3482 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-01-28T15:02:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:35Z Disabled control of system clock
I20260128 15:02:35.360213 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:45613
--webserver_interface=127.13.88.190
--webserver_port=0
--builtin_ntp_servers=127.13.88.148:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:45613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260128 15:02:35.463665 18884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:35.463887 18884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:35.463949 18884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:35.467119 18884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:35.467173 18884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:35.467188 18884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:35.467206 18884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:35.467221 18884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:02:35.470903 18884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:45613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:45613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:35.471850 18884 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:35.472896 18884 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:35.478106 18892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:35.478118 18889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:35.478121 18890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:35.478217 18884 server_base.cc:1060] running on GCE node
I20260128 15:02:35.478871 18884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:35.479708 18884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:35.480844 18884 hybrid_clock.cc:648] HybridClock initialized: now 1769612555480825 us; error 34 us; skew 500 ppm
I20260128 15:02:35.482535 18884 webserver.cc:492] Webserver started at http://127.13.88.190:36367/ using document root <none> and password file <none>
I20260128 15:02:35.483036 18884 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:35.483084 18884 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:35.483228 18884 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:35.484874 18884 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "36e24ffa68dd414dada49ec5ffad3f4b"
format_stamp: "Formatted at 2026-01-28 15:02:35 on dist-test-slave-m28t"
server_key: "70f389b6e91459934d39d8a5c3391fff"
server_key_iv: "24aefa413274b4749a76af694a871732"
server_key_version: "encryptionkey@0"
I20260128 15:02:35.485291 18884 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "36e24ffa68dd414dada49ec5ffad3f4b"
format_stamp: "Formatted at 2026-01-28 15:02:35 on dist-test-slave-m28t"
server_key: "70f389b6e91459934d39d8a5c3391fff"
server_key_iv: "24aefa413274b4749a76af694a871732"
server_key_version: "encryptionkey@0"
I20260128 15:02:35.488355 18884 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260128 15:02:35.490442 18898 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:35.491371 18884 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:35.491461 18884 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "36e24ffa68dd414dada49ec5ffad3f4b"
format_stamp: "Formatted at 2026-01-28 15:02:35 on dist-test-slave-m28t"
server_key: "70f389b6e91459934d39d8a5c3391fff"
server_key_iv: "24aefa413274b4749a76af694a871732"
server_key_version: "encryptionkey@0"
I20260128 15:02:35.491530 18884 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:35.495846 18884 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:35.496316 18884 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:35.496493 18884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:35.502771 18884 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:45613
I20260128 15:02:35.502779 18950 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:45613 every 8 connection(s)
I20260128 15:02:35.503672 18884 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:35.504655 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 18884
I20260128 15:02:35.504760 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:35.504971 13666 external_mini_cluster.cc:1468] Setting key 5ad9a39cc33e73b96713f28fe91335d5
I20260128 15:02:35.506515 18951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:35.509644 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:35.506073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55458 (local address 127.13.88.190:45613)
0128 15:02:35.506605 (+ 532us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:35.506615 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:35.506652 (+ 37us) server_negotiation.cc:408] Connection header received
0128 15:02:35.507241 (+ 589us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:35.507256 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:35.507537 (+ 281us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:35.507800 (+ 263us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:35.508622 (+ 822us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:35.508627 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:35.508636 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:02:35.508646 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:35.508687 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:02:35.508698 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:35.508838 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:35.508899 (+ 61us) server_negotiation.cc:300] Negotiation successful
0128 15:02:35.509110 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":94,"threads_started":1}
I20260128 15:02:35.511878 18951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b: Bootstrap starting.
I20260128 15:02:35.513861 18951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:35.514530 18951 log.cc:826] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:35.516286 18951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b: No bootstrap required, opened a new log
I20260128 15:02:35.518702 18951 raft_consensus.cc:359] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e24ffa68dd414dada49ec5ffad3f4b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 45613 } }
I20260128 15:02:35.518890 18951 raft_consensus.cc:385] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:35.518955 18951 raft_consensus.cc:740] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e24ffa68dd414dada49ec5ffad3f4b, State: Initialized, Role: FOLLOWER
I20260128 15:02:35.519359 18951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e24ffa68dd414dada49ec5ffad3f4b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 45613 } }
I20260128 15:02:35.519482 18951 raft_consensus.cc:399] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:35.519538 18951 raft_consensus.cc:493] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:35.519623 18951 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:35.520380 18951 raft_consensus.cc:515] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e24ffa68dd414dada49ec5ffad3f4b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 45613 } }
I20260128 15:02:35.520723 18951 leader_election.cc:304] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36e24ffa68dd414dada49ec5ffad3f4b; no voters:
I20260128 15:02:35.520991 18951 leader_election.cc:290] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:35.521076 18956 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:35.521260 18956 raft_consensus.cc:697] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [term 1 LEADER]: Becoming Leader. State: Replica: 36e24ffa68dd414dada49ec5ffad3f4b, State: Running, Role: LEADER
I20260128 15:02:35.521602 18956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e24ffa68dd414dada49ec5ffad3f4b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 45613 } }
I20260128 15:02:35.522090 18951 sys_catalog.cc:565] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:35.523157 18958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36e24ffa68dd414dada49ec5ffad3f4b. Latest consensus state: current_term: 1 leader_uuid: "36e24ffa68dd414dada49ec5ffad3f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e24ffa68dd414dada49ec5ffad3f4b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 45613 } } }
I20260128 15:02:35.523149 18957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36e24ffa68dd414dada49ec5ffad3f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e24ffa68dd414dada49ec5ffad3f4b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 45613 } } }
I20260128 15:02:35.523269 18958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:35.523269 18957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:35.523702 18965 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:35.526345 18965 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:35.531152 18965 catalog_manager.cc:1357] Generated new cluster ID: 373a4b42896440ecb670c541f5b56d58
I20260128 15:02:35.531201 18965 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:35.556754 18965 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:35.557049 18965 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:35.563022 18965 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b: Generated new TSK 0
I20260128 15:02:35.563640 18965 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260128 15:02:35.574236 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:45613
--builtin_ntp_servers=127.13.88.148:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260128 15:02:35.676216 18975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:35.676460 18975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:35.676529 18975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:35.679802 18975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:35.679857 18975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:35.679939 18975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:02:35.683674 18975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:45613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:35.684721 18975 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:35.685804 18975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:35.692512 18981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:35.692612 18983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:35.692519 18980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:35.693137 18975 server_base.cc:1060] running on GCE node
I20260128 15:02:35.693498 18975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:35.693992 18975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:35.695142 18975 hybrid_clock.cc:648] HybridClock initialized: now 1769612555695124 us; error 31 us; skew 500 ppm
I20260128 15:02:35.696846 18975 webserver.cc:492] Webserver started at http://127.13.88.129:45875/ using document root <none> and password file <none>
I20260128 15:02:35.697373 18975 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:35.697425 18975 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:35.697589 18975 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:35.699225 18975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "33867ef58b0e43188079f22e146ecfda"
format_stamp: "Formatted at 2026-01-28 15:02:35 on dist-test-slave-m28t"
server_key: "e511cf4ee80714e707a2c94f8a4c6414"
server_key_iv: "739ed7d1d8e3577ae5c8f5deb624fb2c"
server_key_version: "encryptionkey@0"
I20260128 15:02:35.699649 18975 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "33867ef58b0e43188079f22e146ecfda"
format_stamp: "Formatted at 2026-01-28 15:02:35 on dist-test-slave-m28t"
server_key: "e511cf4ee80714e707a2c94f8a4c6414"
server_key_iv: "739ed7d1d8e3577ae5c8f5deb624fb2c"
server_key_version: "encryptionkey@0"
I20260128 15:02:35.702796 18975 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260128 15:02:35.704892 18989 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:35.705822 18975 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:35.705920 18975 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "33867ef58b0e43188079f22e146ecfda"
format_stamp: "Formatted at 2026-01-28 15:02:35 on dist-test-slave-m28t"
server_key: "e511cf4ee80714e707a2c94f8a4c6414"
server_key_iv: "739ed7d1d8e3577ae5c8f5deb624fb2c"
server_key_version: "encryptionkey@0"
I20260128 15:02:35.705999 18975 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:35.710440 18975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:35.710958 18975 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:35.711107 18975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:35.711565 18975 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:35.712450 18975 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:35.712500 18975 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:35.712548 18975 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:35.712574 18975 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:35.720666 18975 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:39595
I20260128 15:02:35.720695 19102 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:39595 every 8 connection(s)
I20260128 15:02:35.721521 18975 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:02:35.726595 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:35.723158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:35087 (local address 127.13.88.190:45613)
0128 15:02:35.723298 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:35.723302 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:35.724366 (+ 1064us) server_negotiation.cc:408] Connection header received
0128 15:02:35.725152 (+ 786us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:35.725155 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:35.725195 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:35.725271 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:35.726232 (+ 961us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:35.726236 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:35.726238 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:02:35.726250 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:35.726271 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:02:35.726279 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:35.726359 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:35.726458 (+ 99us) server_negotiation.cc:300] Negotiation successful
0128 15:02:35.726492 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260128 15:02:35.727274 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:35.723585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:45613 (local address 127.13.88.129:35087)
0128 15:02:35.724235 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:35.724266 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:35.724963 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:35.725399 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:35.725407 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:35.725486 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:35.725998 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 15:02:35.726000 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:35.726033 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 15:02:35.726086 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:35.726354 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:35.726359 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:35.726362 (+ 3us) client_negotiation.cc:769] Sending connection context
0128 15:02:35.726505 (+ 143us) client_negotiation.cc:240] Negotiation successful
0128 15:02:35.726671 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":141,"threads_started":1}
I20260128 15:02:35.728327 19103 heartbeater.cc:344] Connected to a master server at 127.13.88.190:45613
I20260128 15:02:35.728569 19103 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:35.728935 19103 heartbeater.cc:507] Master 127.13.88.190:45613 requested a full tablet report, sending...
I20260128 15:02:35.729239 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 18975
I20260128 15:02:35.729334 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:02:35.729611 13666 external_mini_cluster.cc:1468] Setting key cf3be564c22d3ecd2d88e365a0664e3e
I20260128 15:02:35.730320 18915 ts_manager.cc:194] Registered new tserver with Master: 33867ef58b0e43188079f22e146ecfda (127.13.88.129:39595)
I20260128 15:02:35.731468 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:45613
--builtin_ntp_servers=127.13.88.148:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260128 15:02:35.832731 19106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:35.832968 19106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:35.833045 19106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:35.836313 19106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:35.836365 19106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:35.836483 19106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:02:35.840174 19106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:45613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:35.841221 19106 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:35.842331 19106 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:35.848521 19112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:35.848567 19111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:35.848666 19106 server_base.cc:1060] running on GCE node
W20260128 15:02:35.848528 19114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:35.849120 19106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:35.849610 19106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:35.850755 19106 hybrid_clock.cc:648] HybridClock initialized: now 1769612555850741 us; error 25 us; skew 500 ppm
I20260128 15:02:35.852308 19106 webserver.cc:492] Webserver started at http://127.13.88.130:39203/ using document root <none> and password file <none>
I20260128 15:02:35.852839 19106 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:35.852895 19106 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:35.853087 19106 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:35.854748 19106 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "f2d1d87ebe284c14813fd5bffa466518"
format_stamp: "Formatted at 2026-01-28 15:02:35 on dist-test-slave-m28t"
server_key: "62e3d28e95730c55118089bc20bc0113"
server_key_iv: "6e4b3b019cd4146db320293a14f40011"
server_key_version: "encryptionkey@0"
I20260128 15:02:35.855192 19106 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "f2d1d87ebe284c14813fd5bffa466518"
format_stamp: "Formatted at 2026-01-28 15:02:35 on dist-test-slave-m28t"
server_key: "62e3d28e95730c55118089bc20bc0113"
server_key_iv: "6e4b3b019cd4146db320293a14f40011"
server_key_version: "encryptionkey@0"
I20260128 15:02:35.858302 19106 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260128 15:02:35.860236 19120 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:35.861196 19106 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:35.861291 19106 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "f2d1d87ebe284c14813fd5bffa466518"
format_stamp: "Formatted at 2026-01-28 15:02:35 on dist-test-slave-m28t"
server_key: "62e3d28e95730c55118089bc20bc0113"
server_key_iv: "6e4b3b019cd4146db320293a14f40011"
server_key_version: "encryptionkey@0"
I20260128 15:02:35.861370 19106 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:35.865779 19106 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:35.866297 19106 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:35.866439 19106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:35.866900 19106 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:35.867756 19106 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:35.867803 19106 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:35.867851 19106 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:35.867877 19106 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:35.876176 19106 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:40361
I20260128 15:02:35.876205 19233 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:40361 every 8 connection(s)
I20260128 15:02:35.877054 19106 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
I20260128 15:02:35.881822 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:35.878596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:44295 (local address 127.13.88.190:45613)
0128 15:02:35.878721 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:35.878725 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:35.879553 (+ 828us) server_negotiation.cc:408] Connection header received
0128 15:02:35.880298 (+ 745us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:35.880302 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:35.880346 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:35.880434 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:35.881390 (+ 956us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:35.881395 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:35.881398 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:02:35.881410 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:35.881430 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:02:35.881437 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:35.881511 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:35.881647 (+ 136us) server_negotiation.cc:300] Negotiation successful
0128 15:02:35.881690 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260128 15:02:35.882450 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:35.878854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:45613 (local address 127.13.88.130:44295)
0128 15:02:35.879426 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:35.879457 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:35.880118 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:35.880589 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:35.880597 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:35.880675 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:35.881170 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 15:02:35.881172 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:35.881202 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 15:02:35.881255 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:35.881506 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:35.881513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:35.881518 (+ 5us) client_negotiation.cc:769] Sending connection context
0128 15:02:35.881698 (+ 180us) client_negotiation.cc:240] Negotiation successful
0128 15:02:35.881902 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":92,"threads_started":1}
I20260128 15:02:35.883394 19234 heartbeater.cc:344] Connected to a master server at 127.13.88.190:45613
I20260128 15:02:35.883641 19234 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:35.884071 19234 heartbeater.cc:507] Master 127.13.88.190:45613 requested a full tablet report, sending...
I20260128 15:02:35.885092 18915 ts_manager.cc:194] Registered new tserver with Master: f2d1d87ebe284c14813fd5bffa466518 (127.13.88.130:40361)
I20260128 15:02:35.885739 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 19106
I20260128 15:02:35.885871 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:02:35.886126 13666 external_mini_cluster.cc:1468] Setting key 48c9f8a4bf59267f3baaa3960a962b39
I20260128 15:02:35.887957 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:45613
--builtin_ntp_servers=127.13.88.148:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260128 15:02:35.992359 19237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:35.992578 19237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:35.992625 19237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:35.995810 19237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:35.995859 19237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:35.995930 19237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:02:35.999577 19237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:45613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:36.000574 19237 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:36.001600 19237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:36.007763 19245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:36.007754 19243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:36.007755 19242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:36.007998 19237 server_base.cc:1060] running on GCE node
I20260128 15:02:36.008368 19237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:36.008903 19237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:36.010037 19237 hybrid_clock.cc:648] HybridClock initialized: now 1769612556010010 us; error 38 us; skew 500 ppm
I20260128 15:02:36.011616 19237 webserver.cc:492] Webserver started at http://127.13.88.131:43027/ using document root <none> and password file <none>
I20260128 15:02:36.012152 19237 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:36.012209 19237 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:36.012375 19237 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:36.014007 19237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "a80d5c1e54ed48b0a433d3c8e65fa967"
format_stamp: "Formatted at 2026-01-28 15:02:36 on dist-test-slave-m28t"
server_key: "cc21a6df1012c5371175e41a3defcb33"
server_key_iv: "c823545d9b4c96781434045e197fe201"
server_key_version: "encryptionkey@0"
I20260128 15:02:36.014436 19237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "a80d5c1e54ed48b0a433d3c8e65fa967"
format_stamp: "Formatted at 2026-01-28 15:02:36 on dist-test-slave-m28t"
server_key: "cc21a6df1012c5371175e41a3defcb33"
server_key_iv: "c823545d9b4c96781434045e197fe201"
server_key_version: "encryptionkey@0"
I20260128 15:02:36.017582 19237 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260128 15:02:36.019549 19251 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:36.020493 19237 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:36.020589 19237 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "a80d5c1e54ed48b0a433d3c8e65fa967"
format_stamp: "Formatted at 2026-01-28 15:02:36 on dist-test-slave-m28t"
server_key: "cc21a6df1012c5371175e41a3defcb33"
server_key_iv: "c823545d9b4c96781434045e197fe201"
server_key_version: "encryptionkey@0"
I20260128 15:02:36.020668 19237 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:36.025496 19237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:36.026016 19237 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:36.026156 19237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:36.026620 19237 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:36.027522 19237 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:36.027568 19237 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:36.027617 19237 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:36.027642 19237 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:36.035575 19237 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:41133
I20260128 15:02:36.035614 19364 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:41133 every 8 connection(s)
I20260128 15:02:36.036501 19237 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
I20260128 15:02:36.041237 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.038056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:42119 (local address 127.13.88.190:45613)
0128 15:02:36.038174 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:36.038178 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:36.039003 (+ 825us) server_negotiation.cc:408] Connection header received
0128 15:02:36.039732 (+ 729us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:36.039736 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:36.039781 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:36.039856 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:36.040806 (+ 950us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:36.040810 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:36.040813 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:02:36.040826 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:36.040847 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:02:36.040855 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:36.040931 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:36.041063 (+ 132us) server_negotiation.cc:300] Negotiation successful
0128 15:02:36.041107 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260128 15:02:36.041872 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.038295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:45613 (local address 127.13.88.131:42119)
0128 15:02:36.038873 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:36.038906 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:36.039553 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:36.040002 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:36.040009 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:36.040089 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:36.040592 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 15:02:36.040594 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:36.040625 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 15:02:36.040675 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:36.040923 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:36.040930 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:36.040935 (+ 5us) client_negotiation.cc:769] Sending connection context
0128 15:02:36.041116 (+ 181us) client_negotiation.cc:240] Negotiation successful
0128 15:02:36.041309 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":89,"threads_started":1}
I20260128 15:02:36.042804 19365 heartbeater.cc:344] Connected to a master server at 127.13.88.190:45613
I20260128 15:02:36.043005 19365 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:36.043123 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 19237
I20260128 15:02:36.043212 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:02:36.043385 19365 heartbeater.cc:507] Master 127.13.88.190:45613 requested a full tablet report, sending...
I20260128 15:02:36.043430 13666 external_mini_cluster.cc:1468] Setting key e60b8cf53a38ef1d3b5fce3017c5e119
I20260128 15:02:36.044274 18915 ts_manager.cc:194] Registered new tserver with Master: a80d5c1e54ed48b0a433d3c8e65fa967 (127.13.88.131:41133)
I20260128 15:02:36.045215 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260128 15:02:36.051218 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.046823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.13.88.190:45613)
0128 15:02:36.046939 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:36.046944 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:36.047070 (+ 126us) server_negotiation.cc:408] Connection header received
0128 15:02:36.047158 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:36.047161 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:36.047189 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:36.047249 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:36.050888 (+ 3639us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:36.050891 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:36.050893 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:02:36.050904 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:36.050923 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:02:36.050930 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:36.050995 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:36.051062 (+ 67us) server_negotiation.cc:300] Negotiation successful
0128 15:02:36.051093 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260128 15:02:36.053470 18914 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 15:02:36.055742 18914 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:02:36.065848 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.062629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:39595 (local address 127.0.0.1:48562)
0128 15:02:36.063205 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:36.063244 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:36.063348 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:36.064686 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:36.064689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:36.064703 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:36.065240 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 15:02:36.065242 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:36.065251 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 15:02:36.065271 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:36.065531 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:36.065541 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:36.065545 (+ 4us) client_negotiation.cc:769] Sending connection context
0128 15:02:36.065652 (+ 107us) client_negotiation.cc:240] Negotiation successful
0128 15:02:36.065712 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"spinlock_wait_cycles":7552,"thread_start_us":101,"threads_started":1}
I20260128 15:02:36.065874 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.062755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48562 (local address 127.13.88.129:39595)
0128 15:02:36.063697 (+ 942us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:36.063703 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:36.063727 (+ 24us) server_negotiation.cc:408] Connection header received
0128 15:02:36.063781 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:36.063787 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:36.063944 (+ 157us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:36.064091 (+ 147us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:36.065377 (+ 1286us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:36.065383 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:36.065393 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:02:36.065403 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:36.065447 (+ 44us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:02:36.065456 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:36.065558 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:36.065659 (+ 101us) server_negotiation.cc:300] Negotiation successful
0128 15:02:36.065721 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":121,"threads_started":1}
I20260128 15:02:36.066069 19377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.063288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:40361 (local address 127.0.0.1:41720)
0128 15:02:36.063481 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:36.063493 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:36.063864 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:36.064505 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:36.064511 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:36.064576 (+ 65us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:36.065081 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 15:02:36.065082 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:36.065108 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 15:02:36.065166 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:36.065846 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:36.065849 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:36.065851 (+ 2us) client_negotiation.cc:769] Sending connection context
0128 15:02:36.065902 (+ 51us) client_negotiation.cc:240] Negotiation successful
0128 15:02:36.065936 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":46,"threads_started":1}
I20260128 15:02:36.066360 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.063151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38926 (local address 127.13.88.131:41133)
0128 15:02:36.063694 (+ 543us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:36.063701 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:36.063724 (+ 23us) server_negotiation.cc:408] Connection header received
0128 15:02:36.063777 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:36.063782 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:36.063942 (+ 160us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:36.064076 (+ 134us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:36.065604 (+ 1528us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:36.065610 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:36.065619 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:02:36.065631 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:36.065681 (+ 50us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:02:36.065693 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:36.065793 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:36.066165 (+ 372us) server_negotiation.cc:300] Negotiation successful
0128 15:02:36.066224 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":114,"threads_started":1}
I20260128 15:02:36.066449 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.063582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.13.88.130:40361)
0128 15:02:36.064087 (+ 505us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:36.064093 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:36.064114 (+ 21us) server_negotiation.cc:408] Connection header received
0128 15:02:36.064186 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:36.064191 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:36.064319 (+ 128us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:36.064486 (+ 167us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:36.065657 (+ 1171us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:36.065661 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:36.065667 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:02:36.065677 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:36.065712 (+ 35us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:02:36.065722 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:36.066233 (+ 511us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:36.066271 (+ 38us) server_negotiation.cc:300] Negotiation successful
0128 15:02:36.066318 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":94,"threads_started":1}
I20260128 15:02:36.066095 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.062794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:41133 (local address 127.0.0.1:38926)
0128 15:02:36.063232 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:36.063243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:36.063348 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:36.065093 (+ 1745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:36.065097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:36.065112 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:36.065434 (+ 322us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 15:02:36.065436 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:36.065448 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 15:02:36.065472 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:36.065832 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:36.065836 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:36.065839 (+ 3us) client_negotiation.cc:769] Sending connection context
0128 15:02:36.065947 (+ 108us) client_negotiation.cc:240] Negotiation successful
0128 15:02:36.065973 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"spinlock_wait_cycles":4608,"thread_start_us":59,"threads_started":1}
I20260128 15:02:36.068919 19168 tablet_service.cc:1505] Processing CreateTablet for tablet 802e3a6c510b412380b44dac5fe757a2 (DEFAULT_TABLE table=test-table [id=b980d2edf627441391496ec7c92eca67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:36.069104 19037 tablet_service.cc:1505] Processing CreateTablet for tablet 802e3a6c510b412380b44dac5fe757a2 (DEFAULT_TABLE table=test-table [id=b980d2edf627441391496ec7c92eca67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:36.069104 19299 tablet_service.cc:1505] Processing CreateTablet for tablet 802e3a6c510b412380b44dac5fe757a2 (DEFAULT_TABLE table=test-table [id=b980d2edf627441391496ec7c92eca67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:02:36.069797 19168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 802e3a6c510b412380b44dac5fe757a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:36.069975 19037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 802e3a6c510b412380b44dac5fe757a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:36.069983 19299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 802e3a6c510b412380b44dac5fe757a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:36.074931 19379 tablet_bootstrap.cc:492] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518: Bootstrap starting.
I20260128 15:02:36.076180 19380 tablet_bootstrap.cc:492] T 802e3a6c510b412380b44dac5fe757a2 P 33867ef58b0e43188079f22e146ecfda: Bootstrap starting.
I20260128 15:02:36.076303 19381 tablet_bootstrap.cc:492] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967: Bootstrap starting.
I20260128 15:02:36.077179 19379 tablet_bootstrap.cc:654] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:36.077904 19379 log.cc:826] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:36.078269 19381 tablet_bootstrap.cc:654] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:36.078269 19380 tablet_bootstrap.cc:654] T 802e3a6c510b412380b44dac5fe757a2 P 33867ef58b0e43188079f22e146ecfda: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:36.079072 19381 log.cc:826] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:36.079072 19380 log.cc:826] T 802e3a6c510b412380b44dac5fe757a2 P 33867ef58b0e43188079f22e146ecfda: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:36.080142 19379 tablet_bootstrap.cc:492] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518: No bootstrap required, opened a new log
I20260128 15:02:36.080325 19379 ts_tablet_manager.cc:1403] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260128 15:02:36.080765 19381 tablet_bootstrap.cc:492] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967: No bootstrap required, opened a new log
I20260128 15:02:36.080941 19381 ts_tablet_manager.cc:1403] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260128 15:02:36.081219 19380 tablet_bootstrap.cc:492] T 802e3a6c510b412380b44dac5fe757a2 P 33867ef58b0e43188079f22e146ecfda: No bootstrap required, opened a new log
I20260128 15:02:36.081400 19380 ts_tablet_manager.cc:1403] T 802e3a6c510b412380b44dac5fe757a2 P 33867ef58b0e43188079f22e146ecfda: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260128 15:02:36.082675 19379 raft_consensus.cc:359] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33867ef58b0e43188079f22e146ecfda" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 39595 } } peers { permanent_uuid: "a80d5c1e54ed48b0a433d3c8e65fa967" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 41133 } } peers { permanent_uuid: "f2d1d87ebe284c14813fd5bffa466518" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40361 } }
I20260128 15:02:36.082854 19379 raft_consensus.cc:385] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:36.082923 19379 raft_consensus.cc:740] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d1d87ebe284c14813fd5bffa466518, State: Initialized, Role: FOLLOWER
I20260128 15:02:36.083393 19379 consensus_queue.cc:260] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33867ef58b0e43188079f22e146ecfda" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 39595 } } peers { permanent_uuid: "a80d5c1e54ed48b0a433d3c8e65fa967" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 41133 } } peers { permanent_uuid: "f2d1d87ebe284c14813fd5bffa466518" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40361 } }
I20260128 15:02:36.083856 19381 raft_consensus.cc:359] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33867ef58b0e43188079f22e146ecfda" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 39595 } } peers { permanent_uuid: "a80d5c1e54ed48b0a433d3c8e65fa967" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 41133 } } peers { permanent_uuid: "f2d1d87ebe284c14813fd5bffa466518" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40361 } }
I20260128 15:02:36.084064 19381 raft_consensus.cc:385] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:36.084182 19381 raft_consensus.cc:740] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80d5c1e54ed48b0a433d3c8e65fa967, State: Initialized, Role: FOLLOWER
I20260128 15:02:36.084215 19380 raft_consensus.cc:359] T 802e3a6c510b412380b44dac5fe757a2 P 33867ef58b0e43188079f22e146ecfda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33867ef58b0e43188079f22e146ecfda" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 39595 } } peers { permanent_uuid: "a80d5c1e54ed48b0a433d3c8e65fa967" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 41133 } } peers { permanent_uuid: "f2d1d87ebe284c14813fd5bffa466518" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40361 } }
I20260128 15:02:36.084398 19380 raft_consensus.cc:385] T 802e3a6c510b412380b44dac5fe757a2 P 33867ef58b0e43188079f22e146ecfda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:36.084481 19380 raft_consensus.cc:740] T 802e3a6c510b412380b44dac5fe757a2 P 33867ef58b0e43188079f22e146ecfda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33867ef58b0e43188079f22e146ecfda, State: Initialized, Role: FOLLOWER
I20260128 15:02:36.084739 19379 ts_tablet_manager.cc:1434] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260128 15:02:36.084714 19381 consensus_queue.cc:260] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33867ef58b0e43188079f22e146ecfda" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 39595 } } peers { permanent_uuid: "a80d5c1e54ed48b0a433d3c8e65fa967" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 41133 } } peers { permanent_uuid: "f2d1d87ebe284c14813fd5bffa466518" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40361 } }
I20260128 15:02:36.084815 19380 consensus_queue.cc:260] T 802e3a6c510b412380b44dac5fe757a2 P 33867ef58b0e43188079f22e146ecfda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33867ef58b0e43188079f22e146ecfda" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 39595 } } peers { permanent_uuid: "a80d5c1e54ed48b0a433d3c8e65fa967" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 41133 } } peers { permanent_uuid: "f2d1d87ebe284c14813fd5bffa466518" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40361 } }
I20260128 15:02:36.085095 19234 heartbeater.cc:499] Master 127.13.88.190:45613 was elected leader, sending a full tablet report...
I20260128 15:02:36.085383 19103 heartbeater.cc:499] Master 127.13.88.190:45613 was elected leader, sending a full tablet report...
I20260128 15:02:36.085566 19380 ts_tablet_manager.cc:1434] T 802e3a6c510b412380b44dac5fe757a2 P 33867ef58b0e43188079f22e146ecfda: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260128 15:02:36.085566 19381 ts_tablet_manager.cc:1434] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260128 15:02:36.085860 19365 heartbeater.cc:499] Master 127.13.88.190:45613 was elected leader, sending a full tablet report...
W20260128 15:02:36.127847 19235 tablet.cc:2378] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 15:02:36.202975 19386 raft_consensus.cc:493] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:02:36.203176 19386 raft_consensus.cc:515] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33867ef58b0e43188079f22e146ecfda" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 39595 } } peers { permanent_uuid: "a80d5c1e54ed48b0a433d3c8e65fa967" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 41133 } } peers { permanent_uuid: "f2d1d87ebe284c14813fd5bffa466518" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40361 } }
I20260128 15:02:36.204290 19386 leader_election.cc:290] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33867ef58b0e43188079f22e146ecfda (127.13.88.129:39595), f2d1d87ebe284c14813fd5bffa466518 (127.13.88.130:40361)
I20260128 15:02:36.206082 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.204623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:54723 (local address 127.13.88.129:39595)
0128 15:02:36.204759 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:36.204764 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:36.204924 (+ 160us) server_negotiation.cc:408] Connection header received
0128 15:02:36.204971 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:36.204974 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:36.205019 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:36.205104 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:36.205721 (+ 617us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:36.205724 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:36.205747 (+ 23us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:02:36.205758 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:36.205777 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:02:36.205785 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:36.205853 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:36.205943 (+ 90us) server_negotiation.cc:300] Negotiation successful
0128 15:02:36.205979 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260128 15:02:36.206120 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.204554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:39595 (local address 127.13.88.131:54723)
0128 15:02:36.204748 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:36.204764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:36.204856 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:36.205144 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:36.205148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:36.205160 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:36.205582 (+ 422us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 15:02:36.205584 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:36.205594 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 15:02:36.205618 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:36.205857 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:36.205861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:36.205865 (+ 4us) client_negotiation.cc:769] Sending connection context
0128 15:02:36.205939 (+ 74us) client_negotiation.cc:240] Negotiation successful
0128 15:02:36.205989 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260128 15:02:36.206789 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.204934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:56937 (local address 127.13.88.130:40361)
0128 15:02:36.205100 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:36.205105 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:36.205119 (+ 14us) server_negotiation.cc:408] Connection header received
0128 15:02:36.205227 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:36.205230 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:36.205263 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:36.205335 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:36.206380 (+ 1045us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:36.206385 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:36.206387 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:02:36.206400 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:36.206422 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:02:36.206432 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:36.206512 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:36.206630 (+ 118us) server_negotiation.cc:300] Negotiation successful
0128 15:02:36.206671 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260128 15:02:36.206815 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.204595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:40361 (local address 127.13.88.131:56937)
0128 15:02:36.205049 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:36.205061 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:36.205139 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:36.205359 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:36.205362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:36.205372 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:36.206227 (+ 855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 15:02:36.206229 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:36.206240 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 15:02:36.206262 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:36.206534 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:36.206537 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:36.206539 (+ 2us) client_negotiation.cc:769] Sending connection context
0128 15:02:36.206622 (+ 83us) client_negotiation.cc:240] Negotiation successful
0128 15:02:36.206659 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":171,"threads_started":1}
I20260128 15:02:36.206784 19057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "802e3a6c510b412380b44dac5fe757a2" candidate_uuid: "a80d5c1e54ed48b0a433d3c8e65fa967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33867ef58b0e43188079f22e146ecfda" is_pre_election: true
I20260128 15:02:36.207055 19057 raft_consensus.cc:2468] T 802e3a6c510b412380b44dac5fe757a2 P 33867ef58b0e43188079f22e146ecfda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80d5c1e54ed48b0a433d3c8e65fa967 in term 0.
I20260128 15:02:36.207301 19188 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "802e3a6c510b412380b44dac5fe757a2" candidate_uuid: "a80d5c1e54ed48b0a433d3c8e65fa967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d1d87ebe284c14813fd5bffa466518" is_pre_election: true
I20260128 15:02:36.207506 19252 leader_election.cc:304] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33867ef58b0e43188079f22e146ecfda, a80d5c1e54ed48b0a433d3c8e65fa967; no voters:
I20260128 15:02:36.207553 19188 raft_consensus.cc:2468] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80d5c1e54ed48b0a433d3c8e65fa967 in term 0.
I20260128 15:02:36.207748 19386 raft_consensus.cc:2804] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:02:36.207839 19386 raft_consensus.cc:493] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:02:36.207891 19386 raft_consensus.cc:3060] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:36.208868 19386 raft_consensus.cc:515] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33867ef58b0e43188079f22e146ecfda" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 39595 } } peers { permanent_uuid: "a80d5c1e54ed48b0a433d3c8e65fa967" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 41133 } } peers { permanent_uuid: "f2d1d87ebe284c14813fd5bffa466518" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40361 } }
I20260128 15:02:36.209187 19386 leader_election.cc:290] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [CANDIDATE]: Term 1 election: Requested vote from peers 33867ef58b0e43188079f22e146ecfda (127.13.88.129:39595), f2d1d87ebe284c14813fd5bffa466518 (127.13.88.130:40361)
I20260128 15:02:36.209529 19188 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "802e3a6c510b412380b44dac5fe757a2" candidate_uuid: "a80d5c1e54ed48b0a433d3c8e65fa967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d1d87ebe284c14813fd5bffa466518"
I20260128 15:02:36.209529 19057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "802e3a6c510b412380b44dac5fe757a2" candidate_uuid: "a80d5c1e54ed48b0a433d3c8e65fa967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33867ef58b0e43188079f22e146ecfda"
I20260128 15:02:36.209646 19188 raft_consensus.cc:3060] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:36.209646 19057 raft_consensus.cc:3060] T 802e3a6c510b412380b44dac5fe757a2 P 33867ef58b0e43188079f22e146ecfda [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:36.211274 19057 raft_consensus.cc:2468] T 802e3a6c510b412380b44dac5fe757a2 P 33867ef58b0e43188079f22e146ecfda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80d5c1e54ed48b0a433d3c8e65fa967 in term 1.
I20260128 15:02:36.211274 19188 raft_consensus.cc:2468] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80d5c1e54ed48b0a433d3c8e65fa967 in term 1.
I20260128 15:02:36.211568 19252 leader_election.cc:304] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33867ef58b0e43188079f22e146ecfda, a80d5c1e54ed48b0a433d3c8e65fa967; no voters:
I20260128 15:02:36.211753 19386 raft_consensus.cc:2804] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:36.211962 19386 raft_consensus.cc:697] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [term 1 LEADER]: Becoming Leader. State: Replica: a80d5c1e54ed48b0a433d3c8e65fa967, State: Running, Role: LEADER
I20260128 15:02:36.212303 19386 consensus_queue.cc:237] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33867ef58b0e43188079f22e146ecfda" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 39595 } } peers { permanent_uuid: "a80d5c1e54ed48b0a433d3c8e65fa967" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 41133 } } peers { permanent_uuid: "f2d1d87ebe284c14813fd5bffa466518" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40361 } }
I20260128 15:02:36.215374 18915 catalog_manager.cc:5654] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 reported cstate change: term changed from 0 to 1, leader changed from <none> to a80d5c1e54ed48b0a433d3c8e65fa967 (127.13.88.131). New cstate: current_term: 1 leader_uuid: "a80d5c1e54ed48b0a433d3c8e65fa967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33867ef58b0e43188079f22e146ecfda" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 39595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80d5c1e54ed48b0a433d3c8e65fa967" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 41133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2d1d87ebe284c14813fd5bffa466518" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40361 } health_report { overall_health: UNKNOWN } } }
W20260128 15:02:36.222492 19104 tablet.cc:2378] T 802e3a6c510b412380b44dac5fe757a2 P 33867ef58b0e43188079f22e146ecfda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 15:02:36.253834 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.252447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.13.88.131:41133)
0128 15:02:36.252567 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:36.252571 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:36.252603 (+ 32us) server_negotiation.cc:408] Connection header received
0128 15:02:36.252798 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:36.252801 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:36.252866 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:36.252933 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:36.253501 (+ 568us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:36.253505 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:36.253506 (+ 1us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:02:36.253517 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:36.253537 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:02:36.253545 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:36.253610 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:36.253665 (+ 55us) server_negotiation.cc:300] Negotiation successful
0128 15:02:36.253696 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260128 15:02:36.259325 19188 raft_consensus.cc:1275] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518 [term 1 FOLLOWER]: Refusing update from remote peer a80d5c1e54ed48b0a433d3c8e65fa967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:36.259330 19057 raft_consensus.cc:1275] T 802e3a6c510b412380b44dac5fe757a2 P 33867ef58b0e43188079f22e146ecfda [term 1 FOLLOWER]: Refusing update from remote peer a80d5c1e54ed48b0a433d3c8e65fa967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:02:36.259987 19386 consensus_queue.cc:1048] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33867ef58b0e43188079f22e146ecfda" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:36.260150 19389 consensus_queue.cc:1048] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2d1d87ebe284c14813fd5bffa466518" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:02:36.269651 19394 mvcc.cc:204] Tried to move back new op lower bound from 7248333030431145984 to 7248333030248349696. Current Snapshot: MvccSnapshot[applied={T|T < 7248333030431145984 or (T in {7248333030431145984})}]
I20260128 15:02:36.276957 18913 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:02:36.277113 18913 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:02:36.279295 18913 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 36e24ffa68dd414dada49ec5ffad3f4b: Sending DeleteTablet for 3 replicas of tablet 802e3a6c510b412380b44dac5fe757a2
I20260128 15:02:36.280015 19037 tablet_service.cc:1552] Processing DeleteTablet for tablet 802e3a6c510b412380b44dac5fe757a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:36 UTC) from {username='slave'} at 127.0.0.1:48562
I20260128 15:02:36.280053 19299 tablet_service.cc:1552] Processing DeleteTablet for tablet 802e3a6c510b412380b44dac5fe757a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:36 UTC) from {username='slave'} at 127.0.0.1:38926
I20260128 15:02:36.280217 19168 tablet_service.cc:1552] Processing DeleteTablet for tablet 802e3a6c510b412380b44dac5fe757a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:02:36 UTC) from {username='slave'} at 127.0.0.1:41720
I20260128 15:02:36.280686 19404 tablet_replica.cc:333] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518: stopping tablet replica
I20260128 15:02:36.280705 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 18975
I20260128 15:02:36.281448 19404 raft_consensus.cc:2243] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:02:36.281795 19404 raft_consensus.cc:2272] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:36.282827 19402 tablet_replica.cc:333] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967: stopping tablet replica
I20260128 15:02:36.283054 19402 raft_consensus.cc:2243] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [term 1 LEADER]: Raft consensus shutting down.
I20260128 15:02:36.283314 19404 ts_tablet_manager.cc:1916] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:36.283499 19402 raft_consensus.cc:2272] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:36.284700 19402 ts_tablet_manager.cc:1916] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:36.285877 19404 ts_tablet_manager.cc:1929] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:02:36.285979 19404 log.cc:1199] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-1/wal/wals/802e3a6c510b412380b44dac5fe757a2
I20260128 15:02:36.286289 19404 ts_tablet_manager.cc:1950] T 802e3a6c510b412380b44dac5fe757a2 P f2d1d87ebe284c14813fd5bffa466518: Deleting consensus metadata
I20260128 15:02:36.286952 19402 ts_tablet_manager.cc:1929] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:02:36.287041 19402 log.cc:1199] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1769612519049359-13666-0/minicluster-data/ts-2/wal/wals/802e3a6c510b412380b44dac5fe757a2
I20260128 15:02:36.287302 19402 ts_tablet_manager.cc:1950] T 802e3a6c510b412380b44dac5fe757a2 P a80d5c1e54ed48b0a433d3c8e65fa967: Deleting consensus metadata
I20260128 15:02:36.287545 18902 catalog_manager.cc:4985] TS f2d1d87ebe284c14813fd5bffa466518 (127.13.88.130:40361): tablet 802e3a6c510b412380b44dac5fe757a2 (table test-table [id=b980d2edf627441391496ec7c92eca67]) successfully deleted
I20260128 15:02:36.288245 18902 catalog_manager.cc:4985] TS a80d5c1e54ed48b0a433d3c8e65fa967 (127.13.88.131:41133): tablet 802e3a6c510b412380b44dac5fe757a2 (table test-table [id=b980d2edf627441391496ec7c92eca67]) successfully deleted
W20260128 15:02:36.288745 18899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.88.129:39595 (error 108)
I20260128 15:02:36.289528 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 19106
I20260128 15:02:36.289681 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.289408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:39595 (local address 127.0.0.1:48566)
0128 15:02:36.289537 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:36.289600 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.129:39595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260128 15:02:36.290772 18899 catalog_manager.cc:4712] TS 33867ef58b0e43188079f22e146ecfda (127.13.88.129:39595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 802e3a6c510b412380b44dac5fe757a2: Network error: Client connection negotiation failed: client connection to 127.13.88.129:39595: connect: Connection refused (error 111)
I20260128 15:02:36.294994 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 19237
I20260128 15:02:36.299746 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 18884
2026-01-28T15:02:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (965 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260128 15:02:36.319422 13666 mini_oidc.cc:150] Starting JWKS server
I20260128 15:02:36.320878 13666 webserver.cc:492] Webserver started at https://127.0.0.1:37079/ using document root <none> and password file <none>
I20260128 15:02:36.320997 13666 mini_oidc.cc:186] Starting OIDC Discovery server
I20260128 15:02:36.321210 13666 webserver.cc:492] Webserver started at http://127.0.0.1:43135/ using document root <none> and password file <none>
2026-01-28T15:02:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:36Z Disabled control of system clock
I20260128 15:02:36.328652 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:39001
--webserver_interface=127.13.88.190
--webserver_port=0
--builtin_ntp_servers=127.13.88.148:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:39001
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37079/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/testchainca.cert with env {}
W20260128 15:02:36.430720 19411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:36.430939 19411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:36.430987 19411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:36.434213 19411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:36.434275 19411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:36.434290 19411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:36.434314 19411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:36.434337 19411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260128 15:02:36.434353 19411 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260128 15:02:36.434371 19411 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37079/jwks.json
I20260128 15:02:36.438627 19411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:39001
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:39001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37079/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:36.439635 19411 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:36.440405 19411 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:36.445624 19419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:36.445626 19416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:36.445684 19417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:36.445739 19411 server_base.cc:1060] running on GCE node
I20260128 15:02:36.446408 19411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:36.447252 19411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:36.448388 19411 hybrid_clock.cc:648] HybridClock initialized: now 1769612556448375 us; error 35 us; skew 500 ppm
I20260128 15:02:36.449975 19411 webserver.cc:492] Webserver started at http://127.13.88.190:45459/ using document root <none> and password file <none>
I20260128 15:02:36.450484 19411 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:36.450536 19411 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:36.450707 19411 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:36.452293 19411 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "46041fcd3a0842de90946bb3d80868b4"
format_stamp: "Formatted at 2026-01-28 15:02:36 on dist-test-slave-m28t"
server_key: "ad4f99459942e1011108969c0775fe36"
server_key_iv: "73a0eab52945c477f8536da92f78dbfe"
server_key_version: "encryptionkey@0"
I20260128 15:02:36.452802 19411 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "46041fcd3a0842de90946bb3d80868b4"
format_stamp: "Formatted at 2026-01-28 15:02:36 on dist-test-slave-m28t"
server_key: "ad4f99459942e1011108969c0775fe36"
server_key_iv: "73a0eab52945c477f8536da92f78dbfe"
server_key_version: "encryptionkey@0"
I20260128 15:02:36.455825 19411 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260128 15:02:36.457816 19425 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:36.458734 19411 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:02:36.458839 19411 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "46041fcd3a0842de90946bb3d80868b4"
format_stamp: "Formatted at 2026-01-28 15:02:36 on dist-test-slave-m28t"
server_key: "ad4f99459942e1011108969c0775fe36"
server_key_iv: "73a0eab52945c477f8536da92f78dbfe"
server_key_version: "encryptionkey@0"
I20260128 15:02:36.458918 19411 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:36.476397 19411 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:36.477033 19411 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:36.477185 19411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:36.483862 19477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:39001 every 8 connection(s)
I20260128 15:02:36.483871 19411 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:39001
I20260128 15:02:36.484906 19411 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:36.487546 19478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:36.492901 19478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4: Bootstrap starting.
I20260128 15:02:36.493464 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 19411
I20260128 15:02:36.493569 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:36.493784 13666 external_mini_cluster.cc:1468] Setting key 8765b36fb368cb2b3b22bcb62d5fd41c
I20260128 15:02:36.495277 19478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:36.495954 19478 log.cc:826] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:36.497825 19478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4: No bootstrap required, opened a new log
I20260128 15:02:36.500629 19478 raft_consensus.cc:359] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041fcd3a0842de90946bb3d80868b4" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39001 } }
I20260128 15:02:36.500799 19478 raft_consensus.cc:385] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:36.500845 19478 raft_consensus.cc:740] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46041fcd3a0842de90946bb3d80868b4, State: Initialized, Role: FOLLOWER
I20260128 15:02:36.501178 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.494905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58630 (local address 127.13.88.190:39001)
0128 15:02:36.495535 (+ 630us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:36.495544 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:36.495577 (+ 33us) server_negotiation.cc:408] Connection header received
0128 15:02:36.496134 (+ 557us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:36.496154 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:36.496573 (+ 419us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:36.496876 (+ 303us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:36.497368 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:36.498125 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:36.498951 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:36.499162 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:36.499661 (+ 499us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:36.499679 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:36.499687 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:02:36.500059 (+ 372us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:36.500099 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:02:36.500113 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:36.500188 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:36.500343 (+ 155us) server_negotiation.cc:300] Negotiation successful
0128 15:02:36.500584 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":134,"threads_started":1}
I20260128 15:02:36.501235 19478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041fcd3a0842de90946bb3d80868b4" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39001 } }
I20260128 15:02:36.501363 19478 raft_consensus.cc:399] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:36.501407 19478 raft_consensus.cc:493] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:36.501466 19478 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:36.502211 19478 raft_consensus.cc:515] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041fcd3a0842de90946bb3d80868b4" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39001 } }
I20260128 15:02:36.502554 19478 leader_election.cc:304] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46041fcd3a0842de90946bb3d80868b4; no voters:
I20260128 15:02:36.502856 19478 leader_election.cc:290] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:36.502947 19483 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:36.503136 19483 raft_consensus.cc:697] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [term 1 LEADER]: Becoming Leader. State: Replica: 46041fcd3a0842de90946bb3d80868b4, State: Running, Role: LEADER
I20260128 15:02:36.503424 19483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041fcd3a0842de90946bb3d80868b4" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39001 } }
I20260128 15:02:36.503711 13666 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260128 15:02:36.503938 19478 sys_catalog.cc:565] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:36.504971 19484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46041fcd3a0842de90946bb3d80868b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041fcd3a0842de90946bb3d80868b4" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39001 } } }
I20260128 15:02:36.505106 19484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:36.504971 19485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46041fcd3a0842de90946bb3d80868b4. Latest consensus state: current_term: 1 leader_uuid: "46041fcd3a0842de90946bb3d80868b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46041fcd3a0842de90946bb3d80868b4" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39001 } } }
I20260128 15:02:36.505203 19485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4 [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:36.505457 19492 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:36.508287 19492 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:36.513406 19492 catalog_manager.cc:1357] Generated new cluster ID: 1abe4a20306e4b1f8a89163905cba52a
I20260128 15:02:36.513461 19492 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:36.529670 19492 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:36.530572 19492 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:36.548545 19492 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 46041fcd3a0842de90946bb3d80868b4: Generated new TSK 0
I20260128 15:02:36.549259 19492 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260128 15:02:36.587671 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.564050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58632 (local address 127.13.88.190:39001)
0128 15:02:36.564164 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:36.564168 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:36.564327 (+ 159us) server_negotiation.cc:408] Connection header received
0128 15:02:36.564539 (+ 212us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:36.564543 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:36.564602 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:36.564698 (+ 96us) server_negotiation.cc:227] Negotiated authn=JWT
0128 15:02:36.565041 (+ 343us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:36.565826 (+ 785us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:36.566430 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:36.566642 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:36.566741 (+ 99us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0128 15:02:36.587191 (+ 20450us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0128 15:02:36.587308 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:36.587452 (+ 144us) server_negotiation.cc:300] Negotiation successful
0128 15:02:36.587516 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29,"thread_start_us":158,"threads_started":1}
I20260128 15:02:36.599081 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:36.595198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58648 (local address 127.13.88.190:39001)
0128 15:02:36.595309 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:36.595314 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:36.595437 (+ 123us) server_negotiation.cc:408] Connection header received
0128 15:02:36.595534 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:36.595538 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:36.595586 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:36.595667 (+ 81us) server_negotiation.cc:227] Negotiated authn=JWT
0128 15:02:36.595966 (+ 299us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:36.596782 (+ 816us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:36.597372 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:36.597570 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:36.597620 (+ 50us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0128 15:02:36.598725 (+ 1105us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0128 15:02:36.598948 (+ 223us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58648: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":27}
W20260128 15:02:36.599157 19481 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58648: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20260128 15:02:36.599741 19514 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:02:36.595099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39001 (local address 127.0.0.1:58648)
0128 15:02:36.595369 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:36.595381 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:36.595464 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:36.595671 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:36.595673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:36.595678 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0128 15:02:36.595881 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:36.595886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:36.596887 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:36.596889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:36.597281 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:36.597288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:36.597371 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:36.597390 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0128 15:02:36.598896 (+ 1506us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0128 15:02:36.599021 (+ 125us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.88.190:39001: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
I20260128 15:02:51.608049 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:51.604931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49154 (local address 127.13.88.190:39001)
0128 15:02:51.605169 (+ 238us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:51.605174 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:51.605190 (+ 16us) server_negotiation.cc:408] Connection header received
0128 15:02:51.605299 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:51.605302 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:51.605350 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:51.605424 (+ 74us) server_negotiation.cc:227] Negotiated authn=JWT
0128 15:02:51.605751 (+ 327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:51.606365 (+ 614us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:51.607005 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:51.607138 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:51.607179 (+ 41us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0128 15:02:51.607793 (+ 614us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0128 15:02:51.607932 (+ 139us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49154: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":75,"threads_started":1}
W20260128 15:02:51.608110 19520 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49154: JWT verification failed: token expired
W20260128 15:02:51.608127 19519 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:02:51.604796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39001 (local address 127.0.0.1:49154)
0128 15:02:51.605086 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:51.605098 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:51.605210 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:51.605438 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:51.605441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:51.605446 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0128 15:02:51.605632 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:51.605638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:51.606478 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:51.606481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:51.606903 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:51.606911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:51.607012 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:51.607032 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0128 15:02:51.607883 (+ 851us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0128 15:02:51.607994 (+ 111us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.88.190:39001: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":130,"threads_started":1}
W20260128 15:02:51.610484 19520 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:02:51.609762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49170 (local address 127.13.88.190:39001)
0128 15:02:51.609886 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:51.609890 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:51.609954 (+ 64us) server_negotiation.cc:408] Connection header received
0128 15:02:51.610061 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:51.610064 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:51.610109 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:51.610172 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:51.610386 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49170: BlockingRecv error: recv got EOF from 127.0.0.1:49170 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260128 15:02:51.613595 19530 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.13.88.190:39001
W20260128 15:02:51.614116 19520 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:02:51.613347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49182 (local address 127.13.88.190:39001)
0128 15:02:51.613450 (+ 103us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:51.613453 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:51.613554 (+ 101us) server_negotiation.cc:408] Connection header received
0128 15:02:51.613739 (+ 185us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:51.613742 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:51.613783 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:51.613845 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:51.614026 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49182: BlockingRecv error: recv got EOF from 127.0.0.1:49182 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260128 15:02:51.614274 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 19411
2026-01-28T15:02:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15334 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260128 15:02:51.653813 13666 mini_oidc.cc:150] Starting JWKS server
I20260128 15:02:51.654809 13666 webserver.cc:492] Webserver started at https://127.0.0.1:39551/ using document root <none> and password file <none>
I20260128 15:02:51.654907 13666 mini_oidc.cc:186] Starting OIDC Discovery server
I20260128 15:02:51.655122 13666 webserver.cc:492] Webserver started at http://127.0.0.1:45315/ using document root <none> and password file <none>
2026-01-28T15:02:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:51Z Disabled control of system clock
I20260128 15:02:51.662879 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:36469
--webserver_interface=127.13.88.190
--webserver_port=0
--builtin_ntp_servers=127.13.88.148:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:36469
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39551/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/testchainca.cert with env {}
W20260128 15:02:51.766459 19536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:51.766671 19536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:51.766731 19536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:51.769933 19536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:51.769984 19536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:51.769999 19536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:51.770018 19536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:51.770051 19536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260128 15:02:51.770078 19536 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260128 15:02:51.770105 19536 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39551/jwks.json
I20260128 15:02:51.774200 19536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:36469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:36469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39551/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:51.775125 19536 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:51.775988 19536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:51.781843 19544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:51.781865 19541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:51.781850 19542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:51.781952 19536 server_base.cc:1060] running on GCE node
I20260128 15:02:51.782485 19536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:51.783282 19536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:51.784446 19536 hybrid_clock.cc:648] HybridClock initialized: now 1769612571784432 us; error 24 us; skew 500 ppm
I20260128 15:02:51.786116 19536 webserver.cc:492] Webserver started at http://127.13.88.190:42971/ using document root <none> and password file <none>
I20260128 15:02:51.786646 19536 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:51.786700 19536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:51.786880 19536 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:51.788525 19536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "5264a27ab19e4346949611e36410f529"
format_stamp: "Formatted at 2026-01-28 15:02:51 on dist-test-slave-m28t"
server_key: "13e1c153a084f4631f45b2ccc6481da7"
server_key_iv: "589bf7f8a8f16e279a53fe9e1407a4e3"
server_key_version: "encryptionkey@0"
I20260128 15:02:51.788965 19536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "5264a27ab19e4346949611e36410f529"
format_stamp: "Formatted at 2026-01-28 15:02:51 on dist-test-slave-m28t"
server_key: "13e1c153a084f4631f45b2ccc6481da7"
server_key_iv: "589bf7f8a8f16e279a53fe9e1407a4e3"
server_key_version: "encryptionkey@0"
I20260128 15:02:51.791991 19536 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260128 15:02:51.794018 19550 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:51.794946 19536 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:02:51.795053 19536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "5264a27ab19e4346949611e36410f529"
format_stamp: "Formatted at 2026-01-28 15:02:51 on dist-test-slave-m28t"
server_key: "13e1c153a084f4631f45b2ccc6481da7"
server_key_iv: "589bf7f8a8f16e279a53fe9e1407a4e3"
server_key_version: "encryptionkey@0"
I20260128 15:02:51.795132 19536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:51.806483 19536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:51.807041 19536 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:51.807197 19536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:51.813655 19602 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:36469 every 8 connection(s)
I20260128 15:02:51.813653 19536 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:36469
I20260128 15:02:51.814602 19536 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:51.817308 19603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:51.817374 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 19536
I20260128 15:02:51.817476 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:51.817684 13666 external_mini_cluster.cc:1468] Setting key 39cbeb798aaede49356f98e6ec62378d
I20260128 15:02:51.823750 19603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529: Bootstrap starting.
I20260128 15:02:51.825073 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:51.818700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.13.88.190:36469)
0128 15:02:51.819276 (+ 576us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:51.819284 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:51.819318 (+ 34us) server_negotiation.cc:408] Connection header received
0128 15:02:51.820169 (+ 851us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:51.820191 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:51.820572 (+ 381us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:51.820875 (+ 303us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:51.821380 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:51.822133 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:51.822797 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:51.823068 (+ 271us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:51.823526 (+ 458us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:51.823573 (+ 47us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:51.823582 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:02:51.823981 (+ 399us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:51.824022 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:02:51.824037 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:51.824126 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:51.824283 (+ 157us) server_negotiation.cc:300] Negotiation successful
0128 15:02:51.824533 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":93,"threads_started":1}
I20260128 15:02:51.825825 19603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:51.826596 19603 log.cc:826] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:51.827632 13666 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260128 15:02:51.828521 19603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529: No bootstrap required, opened a new log
I20260128 15:02:51.830896 19603 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5264a27ab19e4346949611e36410f529" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 36469 } }
I20260128 15:02:51.831076 19603 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:51.831110 19603 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5264a27ab19e4346949611e36410f529, State: Initialized, Role: FOLLOWER
I20260128 15:02:51.831468 19603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5264a27ab19e4346949611e36410f529" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 36469 } }
I20260128 15:02:51.831579 19603 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:51.831621 19603 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:51.831682 19603 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:51.832394 19603 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5264a27ab19e4346949611e36410f529" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 36469 } }
I20260128 15:02:51.832713 19603 leader_election.cc:304] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5264a27ab19e4346949611e36410f529; no voters:
I20260128 15:02:51.832962 19603 leader_election.cc:290] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:51.833079 19608 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:51.833287 19608 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [term 1 LEADER]: Becoming Leader. State: Replica: 5264a27ab19e4346949611e36410f529, State: Running, Role: LEADER
I20260128 15:02:51.833591 19608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5264a27ab19e4346949611e36410f529" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 36469 } }
I20260128 15:02:51.834000 19603 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:51.835260 19610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5264a27ab19e4346949611e36410f529. Latest consensus state: current_term: 1 leader_uuid: "5264a27ab19e4346949611e36410f529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5264a27ab19e4346949611e36410f529" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 36469 } } }
I20260128 15:02:51.835372 19610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:51.835413 19609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5264a27ab19e4346949611e36410f529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5264a27ab19e4346949611e36410f529" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 36469 } } }
I20260128 15:02:51.835511 19609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529 [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:51.836321 19617 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:51.838968 19617 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:51.843825 19617 catalog_manager.cc:1357] Generated new cluster ID: da1d60ee9271408d8e217c77b1a29444
I20260128 15:02:51.843888 19617 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:51.869261 19617 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:51.870502 19617 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:51.878216 19617 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 5264a27ab19e4346949611e36410f529: Generated new TSK 0
I20260128 15:02:51.879009 19617 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260128 15:02:51.913872 19633 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260128 15:02:51.913956 19633 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260128 15:02:51.914523 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:51.891024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.13.88.190:36469)
0128 15:02:51.891158 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:51.891162 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:51.891283 (+ 121us) server_negotiation.cc:408] Connection header received
0128 15:02:51.891432 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:51.891434 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:51.891480 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:51.891556 (+ 76us) server_negotiation.cc:227] Negotiated authn=JWT
0128 15:02:51.891945 (+ 389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:51.892502 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:51.893370 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:51.893502 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:51.893584 (+ 82us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0128 15:02:51.914168 (+ 20584us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39551/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0128 15:02:51.914400 (+ 232us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40078: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39551/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260128 15:02:51.914592 19606 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40078: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39551/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260128 15:02:51.914686 19632 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:02:51.890921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:36469 (local address 127.0.0.1:40078)
0128 15:02:51.891202 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:51.891216 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:51.891324 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:51.891568 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:51.891571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:51.891578 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0128 15:02:51.891858 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:51.891866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:51.892604 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:51.892608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:51.893273 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:51.893283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:51.893391 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:51.893414 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0128 15:02:51.914358 (+ 20944us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39551/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0128 15:02:51.914508 (+ 150us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.88.190:36469: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39551/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
I20260128 15:02:51.915247 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 19536
2026-01-28T15:02:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (299 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260128 15:02:51.953768 13666 mini_oidc.cc:150] Starting JWKS server
I20260128 15:02:51.954640 13666 webserver.cc:492] Webserver started at https://127.0.0.1:38687/ using document root <none> and password file <none>
I20260128 15:02:51.954723 13666 mini_oidc.cc:186] Starting OIDC Discovery server
I20260128 15:02:51.954929 13666 webserver.cc:492] Webserver started at http://127.0.0.1:43297/ using document root <none> and password file <none>
2026-01-28T15:02:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:51Z Disabled control of system clock
I20260128 15:02:51.962544 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:43673
--webserver_interface=127.13.88.190
--webserver_port=0
--builtin_ntp_servers=127.13.88.148:39709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:43673
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38687/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260128 15:02:52.062858 19639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:52.063066 19639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:52.063112 19639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:52.066265 19639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:52.066320 19639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:52.066336 19639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:52.066368 19639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:52.066391 19639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260128 15:02:52.066412 19639 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260128 15:02:52.066431 19639 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38687/jwks.json
I20260128 15:02:52.070096 19639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:39709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:43673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:43673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38687/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:52.071009 19639 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:52.072088 19639 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:52.077404 19645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:52.077409 19644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:52.077556 19647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:52.077709 19639 server_base.cc:1060] running on GCE node
I20260128 15:02:52.078171 19639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:52.078991 19639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:52.080134 19639 hybrid_clock.cc:648] HybridClock initialized: now 1769612572080112 us; error 36 us; skew 500 ppm
I20260128 15:02:52.081771 19639 webserver.cc:492] Webserver started at http://127.13.88.190:40449/ using document root <none> and password file <none>
I20260128 15:02:52.082273 19639 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:52.082321 19639 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:52.082464 19639 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:52.084008 19639 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "7f141e7fbedb43ea9df92eaa8fa42649"
format_stamp: "Formatted at 2026-01-28 15:02:52 on dist-test-slave-m28t"
server_key: "120aad108524babbebaf95594e14567f"
server_key_iv: "51892dfadc196dc308fd6e03243ef959"
server_key_version: "encryptionkey@0"
I20260128 15:02:52.084448 19639 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "7f141e7fbedb43ea9df92eaa8fa42649"
format_stamp: "Formatted at 2026-01-28 15:02:52 on dist-test-slave-m28t"
server_key: "120aad108524babbebaf95594e14567f"
server_key_iv: "51892dfadc196dc308fd6e03243ef959"
server_key_version: "encryptionkey@0"
I20260128 15:02:52.087576 19639 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260128 15:02:52.089763 19653 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:52.090670 19639 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:02:52.090773 19639 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "7f141e7fbedb43ea9df92eaa8fa42649"
format_stamp: "Formatted at 2026-01-28 15:02:52 on dist-test-slave-m28t"
server_key: "120aad108524babbebaf95594e14567f"
server_key_iv: "51892dfadc196dc308fd6e03243ef959"
server_key_version: "encryptionkey@0"
I20260128 15:02:52.090862 19639 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:52.113989 19639 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:52.114606 19639 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:52.114761 19639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:52.121467 19705 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:43673 every 8 connection(s)
I20260128 15:02:52.121472 19639 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:43673
I20260128 15:02:52.122473 19639 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:52.125130 19706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:52.126807 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 19639
I20260128 15:02:52.126905 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:52.127086 13666 external_mini_cluster.cc:1468] Setting key 3820873aaf0e9091c185bf73643e7c55
I20260128 15:02:52.131232 19706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649: Bootstrap starting.
I20260128 15:02:52.133494 19706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:52.134115 19706 log.cc:826] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:52.135675 19706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649: No bootstrap required, opened a new log
I20260128 15:02:52.138414 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:52.128241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.13.88.190:43673)
0128 15:02:52.128875 (+ 634us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:52.128886 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:52.128926 (+ 40us) server_negotiation.cc:408] Connection header received
0128 15:02:52.129547 (+ 621us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:52.129572 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:52.129991 (+ 419us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:52.130339 (+ 348us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:52.130780 (+ 441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:52.131896 (+ 1116us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:52.132601 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:52.132858 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:52.136706 (+ 3848us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:52.136725 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:52.136738 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:02:52.137177 (+ 439us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:52.137242 (+ 65us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:02:52.137260 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:52.137363 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:52.137497 (+ 134us) server_negotiation.cc:300] Negotiation successful
0128 15:02:52.137721 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
I20260128 15:02:52.138511 19706 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f141e7fbedb43ea9df92eaa8fa42649" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 43673 } }
I20260128 15:02:52.138708 19706 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:52.138772 19706 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f141e7fbedb43ea9df92eaa8fa42649, State: Initialized, Role: FOLLOWER
I20260128 15:02:52.139215 19706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f141e7fbedb43ea9df92eaa8fa42649" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 43673 } }
I20260128 15:02:52.139357 19706 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:52.139426 19706 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:52.139508 19706 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:52.140380 19706 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f141e7fbedb43ea9df92eaa8fa42649" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 43673 } }
I20260128 15:02:52.140713 19706 leader_election.cc:304] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f141e7fbedb43ea9df92eaa8fa42649; no voters:
I20260128 15:02:52.140959 13666 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260128 15:02:52.141014 19706 leader_election.cc:290] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:52.141080 19711 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:52.141239 19711 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [term 1 LEADER]: Becoming Leader. State: Replica: 7f141e7fbedb43ea9df92eaa8fa42649, State: Running, Role: LEADER
I20260128 15:02:52.141554 19711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f141e7fbedb43ea9df92eaa8fa42649" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 43673 } }
I20260128 15:02:52.142100 19706 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:52.143249 19713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f141e7fbedb43ea9df92eaa8fa42649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f141e7fbedb43ea9df92eaa8fa42649" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 43673 } } }
I20260128 15:02:52.143354 19713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:52.143366 19712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f141e7fbedb43ea9df92eaa8fa42649. Latest consensus state: current_term: 1 leader_uuid: "7f141e7fbedb43ea9df92eaa8fa42649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f141e7fbedb43ea9df92eaa8fa42649" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 43673 } } }
I20260128 15:02:52.143477 19712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649 [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:52.143846 19720 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:52.146467 19720 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:52.151290 19720 catalog_manager.cc:1357] Generated new cluster ID: e346a005b8864943bbcb64f19a54e42a
I20260128 15:02:52.151351 19720 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:52.183921 19720 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:52.185262 19720 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:52.197691 19720 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 7f141e7fbedb43ea9df92eaa8fa42649: Generated new TSK 0
I20260128 15:02:52.198561 19720 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260128 15:02:52.224455 19736 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260128 15:02:52.224522 19736 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260128 15:02:52.225914 19735 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:02:52.207939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:43673 (local address 127.0.0.1:59350)
0128 15:02:52.208227 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:52.208245 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:52.208337 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:52.208623 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:52.208626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:52.208632 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0128 15:02:52.208829 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:52.208834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:52.209800 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:52.209802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:52.210199 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:52.210206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:52.210299 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:52.210317 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0128 15:02:52.225674 (+ 15357us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38687/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0128 15:02:52.225775 (+ 101us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.88.190:43673: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38687/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":116,"threads_started":1}
I20260128 15:02:52.225951 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:52.208041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.13.88.190:43673)
0128 15:02:52.208161 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:52.208166 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:52.208302 (+ 136us) server_negotiation.cc:408] Connection header received
0128 15:02:52.208460 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:52.208464 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:52.208523 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:52.208625 (+ 102us) server_negotiation.cc:227] Negotiated authn=JWT
0128 15:02:52.208917 (+ 292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:52.209709 (+ 792us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:52.210289 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:52.210485 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:52.210583 (+ 98us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0128 15:02:52.225500 (+ 14917us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38687/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0128 15:02:52.225820 (+ 320us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59350: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38687/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":40}
W20260128 15:02:52.226012 19709 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59350: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38687/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260128 15:02:52.226478 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 19639
2026-01-28T15:02:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (310 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-01-28T15:02:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:52Z Disabled control of system clock
I20260128 15:02:52.271260 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:46565
--webserver_interface=127.13.88.190
--webserver_port=0
--builtin_ntp_servers=127.13.88.148:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:46565
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/testchainca.cert with env {}
W20260128 15:02:52.371109 19740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:52.371322 19740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:52.371378 19740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:52.374539 19740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:52.374596 19740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:52.374611 19740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:52.374629 19740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:52.374660 19740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260128 15:02:52.374691 19740 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260128 15:02:52.374716 19740 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260128 15:02:52.378926 19740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:46565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:46565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:52.379891 19740 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:52.380697 19740 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:52.385814 19745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:52.385807 19746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:52.386006 19740 server_base.cc:1060] running on GCE node
W20260128 15:02:52.385807 19748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:52.386479 19740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:52.387283 19740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:52.388450 19740 hybrid_clock.cc:648] HybridClock initialized: now 1769612572388430 us; error 32 us; skew 500 ppm
I20260128 15:02:52.389930 19740 webserver.cc:492] Webserver started at http://127.13.88.190:41783/ using document root <none> and password file <none>
I20260128 15:02:52.390414 19740 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:52.390461 19740 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:52.390606 19740 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:52.392112 19740 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "257db229b63b4136bd3400a7ba7004dc"
format_stamp: "Formatted at 2026-01-28 15:02:52 on dist-test-slave-m28t"
server_key: "4b226e0d5caa634dfb18d3aecce8f5f6"
server_key_iv: "8352436c56a366dfa13115bb0e174460"
server_key_version: "encryptionkey@0"
I20260128 15:02:52.392560 19740 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "257db229b63b4136bd3400a7ba7004dc"
format_stamp: "Formatted at 2026-01-28 15:02:52 on dist-test-slave-m28t"
server_key: "4b226e0d5caa634dfb18d3aecce8f5f6"
server_key_iv: "8352436c56a366dfa13115bb0e174460"
server_key_version: "encryptionkey@0"
I20260128 15:02:52.395459 19740 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260128 15:02:52.397395 19754 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:52.398301 19740 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:02:52.398389 19740 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "257db229b63b4136bd3400a7ba7004dc"
format_stamp: "Formatted at 2026-01-28 15:02:52 on dist-test-slave-m28t"
server_key: "4b226e0d5caa634dfb18d3aecce8f5f6"
server_key_iv: "8352436c56a366dfa13115bb0e174460"
server_key_version: "encryptionkey@0"
I20260128 15:02:52.398458 19740 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:52.416635 19740 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:52.417264 19740 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:52.417407 19740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:52.423854 19740 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:46565
I20260128 15:02:52.423863 19806 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:46565 every 8 connection(s)
I20260128 15:02:52.424868 19740 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:52.425724 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 19740
I20260128 15:02:52.425853 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:52.426111 13666 external_mini_cluster.cc:1468] Setting key 6108442776804967d132f984e6c2dfdc
I20260128 15:02:52.428628 19807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:52.433475 19807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc: Bootstrap starting.
I20260128 15:02:52.433756 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:52.427239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.13.88.190:46565)
0128 15:02:52.427805 (+ 566us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:52.427813 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:52.427846 (+ 33us) server_negotiation.cc:408] Connection header received
0128 15:02:52.428514 (+ 668us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:52.428532 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:52.428887 (+ 355us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:52.429219 (+ 332us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:52.429678 (+ 459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:52.430783 (+ 1105us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:52.431429 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:52.431692 (+ 263us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:52.432254 (+ 562us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:52.432294 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:52.432303 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:02:52.432690 (+ 387us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:52.432729 (+ 39us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:02:52.432744 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:52.432834 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:52.432976 (+ 142us) server_negotiation.cc:300] Negotiation successful
0128 15:02:52.433182 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":99,"threads_started":1}
I20260128 15:02:52.435632 19807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:52.436324 13666 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260128 15:02:52.436458 19807 log.cc:826] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:52.438184 19807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc: No bootstrap required, opened a new log
I20260128 15:02:52.440507 19807 raft_consensus.cc:359] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257db229b63b4136bd3400a7ba7004dc" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 46565 } }
I20260128 15:02:52.440677 19807 raft_consensus.cc:385] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:52.440729 19807 raft_consensus.cc:740] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257db229b63b4136bd3400a7ba7004dc, State: Initialized, Role: FOLLOWER
I20260128 15:02:52.441112 19807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257db229b63b4136bd3400a7ba7004dc" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 46565 } }
I20260128 15:02:52.441224 19807 raft_consensus.cc:399] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:52.441282 19807 raft_consensus.cc:493] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:52.441365 19807 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:52.442111 19807 raft_consensus.cc:515] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257db229b63b4136bd3400a7ba7004dc" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 46565 } }
I20260128 15:02:52.442407 19807 leader_election.cc:304] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 257db229b63b4136bd3400a7ba7004dc; no voters:
I20260128 15:02:52.442657 19807 leader_election.cc:290] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:52.442754 19812 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:52.442952 19812 raft_consensus.cc:697] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [term 1 LEADER]: Becoming Leader. State: Replica: 257db229b63b4136bd3400a7ba7004dc, State: Running, Role: LEADER
I20260128 15:02:52.443238 19812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257db229b63b4136bd3400a7ba7004dc" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 46565 } }
I20260128 15:02:52.443744 19807 sys_catalog.cc:565] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:52.444780 19814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 257db229b63b4136bd3400a7ba7004dc. Latest consensus state: current_term: 1 leader_uuid: "257db229b63b4136bd3400a7ba7004dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257db229b63b4136bd3400a7ba7004dc" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 46565 } } }
I20260128 15:02:52.444891 19814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:52.444903 19813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "257db229b63b4136bd3400a7ba7004dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257db229b63b4136bd3400a7ba7004dc" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 46565 } } }
I20260128 15:02:52.445003 19813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:52.445345 19821 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:52.448174 19821 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:52.452926 19821 catalog_manager.cc:1357] Generated new cluster ID: aca2564c5320493e9286a2ee19362084
I20260128 15:02:52.452987 19821 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:52.472929 19821 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:52.473819 19821 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:52.492399 19821 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 257db229b63b4136bd3400a7ba7004dc: Generated new TSK 0
I20260128 15:02:52.493163 19821 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260128 15:02:52.532666 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:52.503923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52220 (local address 127.13.88.190:46565)
0128 15:02:52.504053 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:52.504058 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:52.504151 (+ 93us) server_negotiation.cc:408] Connection header received
0128 15:02:52.504296 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:52.504299 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:52.504358 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:52.504487 (+ 129us) server_negotiation.cc:227] Negotiated authn=JWT
0128 15:02:52.504911 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:52.505710 (+ 799us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:52.506339 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:52.506538 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:52.506636 (+ 98us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0128 15:02:52.532149 (+ 25513us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0128 15:02:52.532499 (+ 350us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52220: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":56}
W20260128 15:02:52.532709 19836 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:02:52.503833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:46565 (local address 127.0.0.1:52220)
0128 15:02:52.504081 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:52.504093 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:52.504181 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:52.504490 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:52.504494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:52.504502 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0128 15:02:52.504786 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:52.504797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:52.505821 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:52.505824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:52.506238 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:52.506246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:52.506353 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:52.506372 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0128 15:02:52.532381 (+ 26009us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0128 15:02:52.532559 (+ 178us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.88.190:46565: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":92,"threads_started":1}
W20260128 15:02:52.532747 19810 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52220: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
I20260128 15:02:52.533454 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 19740
2026-01-28T15:02:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (287 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (117 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (117 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:02:52 dist-test-slave-m28t krb5kdc[19848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:02:52 dist-test-slave-m28t krb5kdc[19848](info): set up 2 sockets
Jan 28 15:02:52 dist-test-slave-m28t krb5kdc[19848](info): commencing operation
krb5kdc: starting...
W20260128 15:02:54.596758 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.809s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Jan 28 15:02:54 dist-test-slave-m28t krb5kdc[19848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:02:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:54.724839 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:43777
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:43777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:54.832868 19864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:54.833088 19864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:54.833141 19864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:54.836376 19864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:54.836465 19864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:54.836493 19864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:54.836519 19864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:54.836542 19864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:02:54.841080 19864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:43777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:43777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:54.842139 19864 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:54.842953 19864 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:54.848388 19869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:54.848392 19870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:54.848409 19872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:54.848552 19864 server_base.cc:1060] running on GCE node
I20260128 15:02:54.849210 19864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:54.850155 19864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:54.851311 19864 hybrid_clock.cc:648] HybridClock initialized: now 1769612574851297 us; error 27 us; skew 500 ppm
Jan 28 15:02:54 dist-test-slave-m28t krb5kdc[19848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612574, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:54.853842 19864 init.cc:377] Logged in from keytab as kudu/127.13.88.190@KRBTEST.COM (short username kudu)
I20260128 15:02:54.854912 19864 webserver.cc:492] Webserver started at http://127.13.88.190:34843/ using document root <none> and password file <none>
I20260128 15:02:54.855468 19864 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:54.855520 19864 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:54.855700 19864 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:54.857481 19864 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "3c36de1249df41cd9e45ac8172b9f966"
format_stamp: "Formatted at 2026-01-28 15:02:54 on dist-test-slave-m28t"
server_key: "f5335605431e01004aada74973ac3173"
server_key_iv: "2b5bd9f90688abf8570a8c8f99abf826"
server_key_version: "encryptionkey@0"
I20260128 15:02:54.857967 19864 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "3c36de1249df41cd9e45ac8172b9f966"
format_stamp: "Formatted at 2026-01-28 15:02:54 on dist-test-slave-m28t"
server_key: "f5335605431e01004aada74973ac3173"
server_key_iv: "2b5bd9f90688abf8570a8c8f99abf826"
server_key_version: "encryptionkey@0"
I20260128 15:02:54.861433 19864 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260128 15:02:54.863562 19879 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:54.864642 19864 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260128 15:02:54.864753 19864 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "3c36de1249df41cd9e45ac8172b9f966"
format_stamp: "Formatted at 2026-01-28 15:02:54 on dist-test-slave-m28t"
server_key: "f5335605431e01004aada74973ac3173"
server_key_iv: "2b5bd9f90688abf8570a8c8f99abf826"
server_key_version: "encryptionkey@0"
I20260128 15:02:54.864850 19864 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:54.895537 19864 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:54.898375 19864 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:54.898564 19864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:54.905423 19931 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:43777 every 8 connection(s)
I20260128 15:02:54.905423 19864 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:43777
I20260128 15:02:54.906471 19864 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:54.909269 19932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:54.910450 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 19864
I20260128 15:02:54.910589 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:54.910854 13666 external_mini_cluster.cc:1468] Setting key df197c2f69342b2a60878d6359861b59
I20260128 15:02:54.915374 19932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966: Bootstrap starting.
I20260128 15:02:54.917673 19932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966: Neither blocks nor log segments found. Creating new log.
Jan 28 15:02:54 dist-test-slave-m28t krb5kdc[19848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:54.918543 19932 log.cc:826] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:54.920327 19932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966: No bootstrap required, opened a new log
I20260128 15:02:54.922811 19932 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36de1249df41cd9e45ac8172b9f966" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 43777 } }
I20260128 15:02:54.923000 19932 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:54.923045 19932 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c36de1249df41cd9e45ac8172b9f966, State: Initialized, Role: FOLLOWER
I20260128 15:02:54.923089 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:54.912190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37018 (local address 127.13.88.190:43777)
0128 15:02:54.912688 (+ 498us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:54.912698 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:54.912738 (+ 40us) server_negotiation.cc:408] Connection header received
0128 15:02:54.913289 (+ 551us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:54.913306 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:54.913611 (+ 305us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:54.913922 (+ 311us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:54.914823 (+ 901us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:54.915620 (+ 797us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:54.916282 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:54.916590 (+ 308us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:54.918972 (+ 2382us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:54.918992 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:54.919003 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:54.919043 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:54.920719 (+ 1676us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:54.921114 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:54.921119 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:54.921124 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:54.921200 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:54.921409 (+ 209us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:54.921413 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:54.921415 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:54.921761 (+ 346us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:54.921965 (+ 204us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:54.922218 (+ 253us) server_negotiation.cc:300] Negotiation successful
0128 15:02:54.922473 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":159,"threads_started":1}
I20260128 15:02:54.923470 19932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36de1249df41cd9e45ac8172b9f966" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 43777 } }
I20260128 15:02:54.923599 19932 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:54.923658 19932 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:54.923746 19932 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:54.924754 19932 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36de1249df41cd9e45ac8172b9f966" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 43777 } }
I20260128 15:02:54.925132 19932 leader_election.cc:304] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c36de1249df41cd9e45ac8172b9f966; no voters:
I20260128 15:02:54.925467 19932 leader_election.cc:290] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:54.925559 19937 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:54.925752 19937 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [term 1 LEADER]: Becoming Leader. State: Replica: 3c36de1249df41cd9e45ac8172b9f966, State: Running, Role: LEADER
I20260128 15:02:54.926082 19937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36de1249df41cd9e45ac8172b9f966" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 43777 } }
I20260128 15:02:54.926541 19932 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:54.927713 19939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c36de1249df41cd9e45ac8172b9f966. Latest consensus state: current_term: 1 leader_uuid: "3c36de1249df41cd9e45ac8172b9f966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36de1249df41cd9e45ac8172b9f966" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 43777 } } }
I20260128 15:02:54.927748 19938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c36de1249df41cd9e45ac8172b9f966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c36de1249df41cd9e45ac8172b9f966" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 43777 } } }
I20260128 15:02:54.927907 19938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:54.927906 19939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966 [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:54.928288 19946 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:54.931386 19946 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:54.936902 19946 catalog_manager.cc:1357] Generated new cluster ID: 640ca4b7474546b39358f64bcbf0d75d
I20260128 15:02:54.936971 19946 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:54.959630 19946 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:54.960668 19946 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:54.968322 19946 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 3c36de1249df41cd9e45ac8172b9f966: Generated new TSK 0
I20260128 15:02:54.969112 19946 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:55.018700 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:43777
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:55.129709 19960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:55.129961 19960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:55.130023 19960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:55.133538 19960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:55.133612 19960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:55.133697 19960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:02:55.138211 19960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:43777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:55.139302 19960 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:55.140149 19960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:55.146942 19966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:55.146986 19968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:55.146942 19965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:55.147048 19960 server_base.cc:1060] running on GCE node
I20260128 15:02:55.147552 19960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:55.148077 19960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:55.149211 19960 hybrid_clock.cc:648] HybridClock initialized: now 1769612575149186 us; error 42 us; skew 500 ppm
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[19848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612575, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:55.151778 19960 init.cc:377] Logged in from keytab as kudu/127.13.88.129@KRBTEST.COM (short username kudu)
I20260128 15:02:55.152885 19960 webserver.cc:492] Webserver started at http://127.13.88.129:42561/ using document root <none> and password file <none>
I20260128 15:02:55.153419 19960 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:55.153468 19960 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:55.153661 19960 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:55.155336 19960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "35280bf65eaf455cbdc4deda3946cc28"
format_stamp: "Formatted at 2026-01-28 15:02:55 on dist-test-slave-m28t"
server_key: "981b0a62ae70d7ab0aae01c7034ae06f"
server_key_iv: "46d8bba44492dffc9c8993d5f9d1b62e"
server_key_version: "encryptionkey@0"
I20260128 15:02:55.155791 19960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "35280bf65eaf455cbdc4deda3946cc28"
format_stamp: "Formatted at 2026-01-28 15:02:55 on dist-test-slave-m28t"
server_key: "981b0a62ae70d7ab0aae01c7034ae06f"
server_key_iv: "46d8bba44492dffc9c8993d5f9d1b62e"
server_key_version: "encryptionkey@0"
I20260128 15:02:55.159153 19960 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260128 15:02:55.161365 19975 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:55.162379 19960 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260128 15:02:55.162468 19960 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "35280bf65eaf455cbdc4deda3946cc28"
format_stamp: "Formatted at 2026-01-28 15:02:55 on dist-test-slave-m28t"
server_key: "981b0a62ae70d7ab0aae01c7034ae06f"
server_key_iv: "46d8bba44492dffc9c8993d5f9d1b62e"
server_key_version: "encryptionkey@0"
I20260128 15:02:55.162556 19960 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:55.187487 19960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:55.190533 19960 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:55.190694 19960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:55.191217 19960 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:55.192107 19960 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:55.192157 19960 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:55.192195 19960 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:55.192212 19960 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:55.201071 19960 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:34485
I20260128 15:02:55.201102 20088 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:34485 every 8 connection(s)
I20260128 15:02:55.202021 19960 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:02:55.204582 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 19960
I20260128 15:02:55.204692 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:02:55.204941 13666 external_mini_cluster.cc:1468] Setting key b2312048845afd8120842bed2960ca45
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[19848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612575, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.88.130@KRBTEST.COM; defaulting to no policy
I20260128 15:02:55.215339 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:55.203753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:43561 (local address 127.13.88.190:43777)
0128 15:02:55.203888 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:55.203892 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:55.204647 (+ 755us) server_negotiation.cc:408] Connection header received
0128 15:02:55.205656 (+ 1009us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:55.205660 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:55.205716 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:55.205786 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:55.207771 (+ 1985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.208343 (+ 572us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:55.209207 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.209360 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:55.211862 (+ 2502us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:55.211885 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:55.211887 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:55.211924 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:55.213390 (+ 1466us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:55.213938 (+ 548us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:55.213942 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:55.213944 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:55.214013 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:55.214297 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:55.214304 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:55.214306 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:55.214481 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:55.214560 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:55.215077 (+ 517us) server_negotiation.cc:300] Negotiation successful
0128 15:02:55.215203 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260128 15:02:55.215988 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:55.204012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:43777 (local address 127.13.88.129:43561)
0128 15:02:55.204489 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:55.204539 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:55.205458 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:55.205951 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:55.205959 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:55.206386 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:55.207608 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:55.207627 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.208495 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:55.208499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:55.209062 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:55.209073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.209306 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:55.209920 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:55.209963 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:55.211686 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:55.213521 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:55.213528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:55.213544 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:55.213805 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:55.214118 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:55.214120 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:55.214122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:55.214222 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:55.214562 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:55.214567 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:55.214829 (+ 262us) client_negotiation.cc:769] Sending connection context
0128 15:02:55.215009 (+ 180us) client_negotiation.cc:240] Negotiation successful
0128 15:02:55.215243 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.130@KRBTEST.COM" created.
I20260128 15:02:55.217270 20089 heartbeater.cc:344] Connected to a master server at 127.13.88.190:43777
I20260128 15:02:55.217514 20089 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:55.218103 20089 heartbeater.cc:507] Master 127.13.88.190:43777 requested a full tablet report, sending...
I20260128 15:02:55.219755 19896 ts_manager.cc:194] Registered new tserver with Master: 35280bf65eaf455cbdc4deda3946cc28 (127.13.88.129:34485)
I20260128 15:02:55.221007 19896 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.129@KRBTEST.COM'} at 127.13.88.129:43561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:55.251760 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:43777
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:55.357509 20096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:55.357738 20096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:55.357816 20096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:55.361110 20096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:55.361176 20096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:55.361274 20096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:02:55.365499 20096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:43777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:55.366588 20096 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:55.367408 20096 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:55.373805 20102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:55.374030 20096 server_base.cc:1060] running on GCE node
W20260128 15:02:55.373807 20101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:55.373832 20104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:55.374428 20096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:55.374948 20096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:55.376101 20096 hybrid_clock.cc:648] HybridClock initialized: now 1769612575376088 us; error 26 us; skew 500 ppm
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[19848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612575, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:55.378598 20096 init.cc:377] Logged in from keytab as kudu/127.13.88.130@KRBTEST.COM (short username kudu)
I20260128 15:02:55.379617 20096 webserver.cc:492] Webserver started at http://127.13.88.130:42935/ using document root <none> and password file <none>
I20260128 15:02:55.380128 20096 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:55.380177 20096 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:55.380362 20096 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:55.382005 20096 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "102bf9a70eb04e2ea0afc1d49cc32ca4"
format_stamp: "Formatted at 2026-01-28 15:02:55 on dist-test-slave-m28t"
server_key: "c35255d49757009786837d621330ee48"
server_key_iv: "4e91859a35a0e6fe42b7a9f8449c5ff1"
server_key_version: "encryptionkey@0"
I20260128 15:02:55.382454 20096 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "102bf9a70eb04e2ea0afc1d49cc32ca4"
format_stamp: "Formatted at 2026-01-28 15:02:55 on dist-test-slave-m28t"
server_key: "c35255d49757009786837d621330ee48"
server_key_iv: "4e91859a35a0e6fe42b7a9f8449c5ff1"
server_key_version: "encryptionkey@0"
I20260128 15:02:55.385721 20096 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260128 15:02:55.387753 20111 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:55.388746 20096 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260128 15:02:55.388844 20096 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "102bf9a70eb04e2ea0afc1d49cc32ca4"
format_stamp: "Formatted at 2026-01-28 15:02:55 on dist-test-slave-m28t"
server_key: "c35255d49757009786837d621330ee48"
server_key_iv: "4e91859a35a0e6fe42b7a9f8449c5ff1"
server_key_version: "encryptionkey@0"
I20260128 15:02:55.388926 20096 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:55.404114 20096 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:55.406947 20096 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:55.407132 20096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:55.407696 20096 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:55.408636 20096 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:55.408687 20096 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:55.408739 20096 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:55.408766 20096 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:55.417467 20096 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:41239
I20260128 15:02:55.417500 20224 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:41239 every 8 connection(s)
I20260128 15:02:55.418426 20096 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[19848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612575, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:55.426918 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 20096
I20260128 15:02:55.427013 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:02:55.427290 13666 external_mini_cluster.cc:1468] Setting key e9787ffebd7d2abdaca95748391ac462
I20260128 15:02:55.430581 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:55.420119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:40311 (local address 127.13.88.190:43777)
0128 15:02:55.420289 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:55.420295 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:55.421033 (+ 738us) server_negotiation.cc:408] Connection header received
0128 15:02:55.421838 (+ 805us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:55.421841 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:55.421891 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:55.421962 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:55.423615 (+ 1653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.424126 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:55.424733 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.424867 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:55.426881 (+ 2014us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:55.426902 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:55.426904 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:55.426930 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:55.428482 (+ 1552us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:55.428927 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:55.428931 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:55.428933 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:55.428976 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:55.429281 (+ 305us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:55.429285 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:55.429287 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:55.429466 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:55.429557 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:55.430198 (+ 641us) server_negotiation.cc:300] Negotiation successful
0128 15:02:55.430389 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260128 15:02:55.431186 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:55.420456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:43777 (local address 127.13.88.130:40311)
0128 15:02:55.420889 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:55.420925 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:55.421665 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:55.422107 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:55.422115 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:55.422515 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:55.423462 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:55.423479 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.424218 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:55.424220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:55.424654 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:55.424661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.424828 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:55.425353 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:55.425378 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:55.426737 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:55.428583 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:55.428589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:55.428602 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:55.428828 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:55.429051 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:55.429053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:55.429055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:55.429142 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:55.429582 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:55.429590 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:55.429894 (+ 304us) client_negotiation.cc:769] Sending connection context
0128 15:02:55.430126 (+ 232us) client_negotiation.cc:240] Negotiation successful
0128 15:02:55.430399 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
I20260128 15:02:55.432291 20225 heartbeater.cc:344] Connected to a master server at 127.13.88.190:43777
I20260128 15:02:55.432598 20225 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:55.433219 20225 heartbeater.cc:507] Master 127.13.88.190:43777 requested a full tablet report, sending...
I20260128 15:02:55.434346 19896 ts_manager.cc:194] Registered new tserver with Master: 102bf9a70eb04e2ea0afc1d49cc32ca4 (127.13.88.130:41239)
I20260128 15:02:55.435021 19896 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.130@KRBTEST.COM'} at 127.13.88.130:40311
WARNING: no policy specified for kudu/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:02:55.472637 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:43777
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:55.578576 20232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:55.578809 20232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:55.578873 20232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:55.582170 20232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:55.582237 20232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:55.582327 20232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:02:55.586634 20232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:43777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:55.587728 20232 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:55.588613 20232 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:55.595045 20237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:55.595214 20232 server_base.cc:1060] running on GCE node
W20260128 15:02:55.595317 20238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:55.595343 20240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:55.595805 20232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:55.596372 20232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:55.597568 20232 hybrid_clock.cc:648] HybridClock initialized: now 1769612575597553 us; error 29 us; skew 500 ppm
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[19848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612575, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:55.600191 20232 init.cc:377] Logged in from keytab as kudu/127.13.88.131@KRBTEST.COM (short username kudu)
I20260128 15:02:55.601315 20232 webserver.cc:492] Webserver started at http://127.13.88.131:41461/ using document root <none> and password file <none>
I20260128 15:02:55.601859 20232 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:55.601909 20232 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:55.602106 20232 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:55.603834 20232 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "5e0e6335796b459fa494743bcb0d1bc8"
format_stamp: "Formatted at 2026-01-28 15:02:55 on dist-test-slave-m28t"
server_key: "e9081a5c4318d447beade699bc428f9c"
server_key_iv: "d74fb41c03730e3c48f23c6d7fc99f72"
server_key_version: "encryptionkey@0"
I20260128 15:02:55.604307 20232 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "5e0e6335796b459fa494743bcb0d1bc8"
format_stamp: "Formatted at 2026-01-28 15:02:55 on dist-test-slave-m28t"
server_key: "e9081a5c4318d447beade699bc428f9c"
server_key_iv: "d74fb41c03730e3c48f23c6d7fc99f72"
server_key_version: "encryptionkey@0"
I20260128 15:02:55.607707 20232 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260128 15:02:55.609974 20247 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:55.611047 20232 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:02:55.611157 20232 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "5e0e6335796b459fa494743bcb0d1bc8"
format_stamp: "Formatted at 2026-01-28 15:02:55 on dist-test-slave-m28t"
server_key: "e9081a5c4318d447beade699bc428f9c"
server_key_iv: "d74fb41c03730e3c48f23c6d7fc99f72"
server_key_version: "encryptionkey@0"
I20260128 15:02:55.611248 20232 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:55.629197 20232 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:55.632107 20232 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:55.632297 20232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:55.632917 20232 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:55.633841 20232 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:55.633893 20232 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:55.633944 20232 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:55.633971 20232 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:55.642602 20232 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:37377
I20260128 15:02:55.642633 20360 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:37377 every 8 connection(s)
I20260128 15:02:55.643576 20232 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
I20260128 15:02:55.648002 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 20232
I20260128 15:02:55.648131 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:02:55.648391 13666 external_mini_cluster.cc:1468] Setting key c32230766932fe6d9487ccb39668a5b6
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[19848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612575, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:02:55.655565 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:55.645437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:59789 (local address 127.13.88.190:43777)
0128 15:02:55.645610 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:55.645615 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:55.646255 (+ 640us) server_negotiation.cc:408] Connection header received
0128 15:02:55.647088 (+ 833us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:55.647091 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:55.647142 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:55.647218 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:55.649112 (+ 1894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.649609 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:55.650228 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.650410 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:55.652400 (+ 1990us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:55.652443 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:55.652447 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:55.652475 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:55.653737 (+ 1262us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:55.654194 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:55.654198 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:55.654199 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:55.654252 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:55.654501 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:55.654504 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:55.654506 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:55.654664 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:55.654746 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:55.655253 (+ 507us) server_negotiation.cc:300] Negotiation successful
0128 15:02:55.655394 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260128 15:02:55.656184 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:55.645693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:43777 (local address 127.13.88.131:59789)
0128 15:02:55.646118 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:55.646153 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:55.646885 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:55.647376 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:55.647383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:55.647773 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:55.648968 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:55.648983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.649720 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:55.649723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:55.650125 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:55.650132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.650337 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:55.650875 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:55.650899 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:55.652263 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:55.653851 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:55.653857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:55.653869 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:55.654102 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:55.654329 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:55.654331 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:55.654333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:55.654420 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:55.654759 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:55.654767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:55.655005 (+ 238us) client_negotiation.cc:769] Sending connection context
0128 15:02:55.655205 (+ 200us) client_negotiation.cc:240] Negotiation successful
0128 15:02:55.655505 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":115,"threads_started":1}
I20260128 15:02:55.657306 20361 heartbeater.cc:344] Connected to a master server at 127.13.88.190:43777
I20260128 15:02:55.657522 20361 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:55.658002 20361 heartbeater.cc:507] Master 127.13.88.190:43777 requested a full tablet report, sending...
I20260128 15:02:55.658974 19896 ts_manager.cc:194] Registered new tserver with Master: 5e0e6335796b459fa494743bcb0d1bc8 (127.13.88.131:37377)
I20260128 15:02:55.659510 19896 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.131@KRBTEST.COM'} at 127.13.88.131:59789
I20260128 15:02:55.661991 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/test-admin.keytab.
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[19848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:55.675796 13666 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260128 15:02:55.678584 20368 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:02:55.677317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.13.88.129:34485)
0128 15:02:55.677592 (+ 275us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:55.677598 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:55.677619 (+ 21us) server_negotiation.cc:408] Connection header received
0128 15:02:55.677666 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:55.677671 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:55.677833 (+ 162us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:55.677966 (+ 133us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:55.678434 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47050: BlockingRecv error: recv got EOF from 127.0.0.1:47050 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":70,"threads_started":1}
W20260128 15:02:55.678714 13666 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/krb5cc' of type FILE
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[19848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:55.679818 13666 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[19848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:55.680797 13666 init.cc:295] Successfully reacquired a new kerberos TGT
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[19848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.129@KRBTEST.COM
I20260128 15:02:55.689957 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:55.681740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47058 (local address 127.13.88.129:34485)
0128 15:02:55.681865 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:55.681869 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:55.681921 (+ 52us) server_negotiation.cc:408] Connection header received
0128 15:02:55.682037 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:55.682041 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:55.682087 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:55.682157 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:55.682761 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.683506 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:55.684213 (+ 707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.684452 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:55.686375 (+ 1923us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:55.686401 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:55.686414 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:55.686441 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:55.688205 (+ 1764us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:55.688661 (+ 456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:55.688668 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:55.688670 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:55.688737 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:55.689073 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:55.689077 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:55.689078 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:55.689303 (+ 225us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:55.689484 (+ 181us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:55.689635 (+ 151us) server_negotiation.cc:300] Negotiation successful
0128 15:02:55.689780 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260128 15:02:55.693748 20368 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:02:55.692749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47066 (local address 127.13.88.129:34485)
0128 15:02:55.692882 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:55.692887 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:55.692994 (+ 107us) server_negotiation.cc:408] Connection header received
0128 15:02:55.693107 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:55.693110 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:55.693154 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:55.693224 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:55.693640 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47066: BlockingRecv error: recv got EOF from 127.0.0.1:47066 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260128 15:02:55.694067 13666 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/krb5cc' of type FILE
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[19848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:55.695068 13666 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[19848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:55.695909 13666 init.cc:295] Successfully reacquired a new kerberos TGT
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[19848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.129@KRBTEST.COM
I20260128 15:02:55.703567 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:55.696834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47082 (local address 127.13.88.129:34485)
0128 15:02:55.696972 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:55.696976 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:55.697045 (+ 69us) server_negotiation.cc:408] Connection header received
0128 15:02:55.697177 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:55.697180 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:55.697228 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:55.697292 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:55.697912 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.698409 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:55.698999 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.699220 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:55.701103 (+ 1883us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:55.701124 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:55.701126 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:55.701151 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:55.702265 (+ 1114us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:55.702603 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:55.702606 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:55.702608 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:55.702649 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:55.702847 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:55.702851 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:55.702853 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:55.702998 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:55.703111 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:55.703320 (+ 209us) server_negotiation.cc:300] Negotiation successful
0128 15:02:55.703421 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260128 15:02:55.706149 20368 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:02:55.705265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.13.88.129:34485)
0128 15:02:55.705386 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:55.705389 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:55.705440 (+ 51us) server_negotiation.cc:408] Connection header received
0128 15:02:55.705521 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:55.705523 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:55.705558 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:55.705620 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:55.706053 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47090: BlockingRecv error: recv got EOF from 127.0.0.1:47090 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260128 15:02:55.706404 13666 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1769612519049359-13666-0/krb5kdc/krb5cc)
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[19848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:55.707386 13666 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[19848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:55.708154 13666 init.cc:295] Successfully reacquired a new kerberos TGT
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[19848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.129@KRBTEST.COM
I20260128 15:02:55.715643 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:55.708935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47098 (local address 127.13.88.129:34485)
0128 15:02:55.709066 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:55.709069 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:55.709108 (+ 39us) server_negotiation.cc:408] Connection header received
0128 15:02:55.709194 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:55.709196 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:55.709236 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:55.709297 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:55.709799 (+ 502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.710268 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:55.710899 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:55.711066 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:55.712993 (+ 1927us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:55.713009 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:55.713011 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:55.713035 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:55.714308 (+ 1273us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:55.714653 (+ 345us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:55.714656 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:55.714657 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:55.714697 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:55.714891 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:55.714894 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:55.714896 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:55.715036 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:55.715159 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:55.715380 (+ 221us) server_negotiation.cc:300] Negotiation successful
0128 15:02:55.715482 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260128 15:02:55.716724 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 19960
I20260128 15:02:55.721877 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 20096
I20260128 15:02:55.726773 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 20232
I20260128 15:02:55.731550 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 19864
2026-01-28T15:02:55Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2964 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[20381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[20381](info): set up 2 sockets
Jan 28 15:02:55 dist-test-slave-m28t krb5kdc[20381](info): commencing operation
krb5kdc: starting...
W20260128 15:02:57.576826 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.824s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Jan 28 15:02:57 dist-test-slave-m28t krb5kdc[20381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:02:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:02:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
I20260128 15:02:57.711345 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:41873
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:41873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:57.824671 20397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:57.824939 20397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:57.825016 20397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:57.828449 20397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:57.828526 20397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:57.828558 20397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:57.828583 20397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:57.828617 20397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:02:57.833113 20397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:41873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:41873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:57.834264 20397 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:57.835184 20397 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:57.841039 20402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:57.841039 20405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:57.841157 20397 server_base.cc:1060] running on GCE node
W20260128 15:02:57.841109 20403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:57.841846 20397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:57.842788 20397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:57.843924 20397 hybrid_clock.cc:648] HybridClock initialized: now 1769612577843903 us; error 38 us; skew 500 ppm
Jan 28 15:02:57 dist-test-slave-m28t krb5kdc[20381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612577, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:57.846525 20397 init.cc:377] Logged in from keytab as oryx/127.13.88.190@KRBTEST.COM (short username oryx)
I20260128 15:02:57.847635 20397 webserver.cc:492] Webserver started at http://127.13.88.190:38297/ using document root <none> and password file <none>
I20260128 15:02:57.848153 20397 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:57.848201 20397 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:57.848390 20397 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:57.850127 20397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "514b33d5af9e48e49897a8adce27696b"
format_stamp: "Formatted at 2026-01-28 15:02:57 on dist-test-slave-m28t"
server_key: "98f6c4adedf1abf76582a171620dd801"
server_key_iv: "689e62cde13656bc06b1d82e745211ff"
server_key_version: "encryptionkey@0"
I20260128 15:02:57.850651 20397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "514b33d5af9e48e49897a8adce27696b"
format_stamp: "Formatted at 2026-01-28 15:02:57 on dist-test-slave-m28t"
server_key: "98f6c4adedf1abf76582a171620dd801"
server_key_iv: "689e62cde13656bc06b1d82e745211ff"
server_key_version: "encryptionkey@0"
I20260128 15:02:57.854290 20397 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260128 15:02:57.856736 20412 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:57.857800 20397 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260128 15:02:57.857925 20397 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "514b33d5af9e48e49897a8adce27696b"
format_stamp: "Formatted at 2026-01-28 15:02:57 on dist-test-slave-m28t"
server_key: "98f6c4adedf1abf76582a171620dd801"
server_key_iv: "689e62cde13656bc06b1d82e745211ff"
server_key_version: "encryptionkey@0"
I20260128 15:02:57.858023 20397 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:57.894505 20397 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:57.900475 20397 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:57.900664 20397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:57.907752 20464 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:41873 every 8 connection(s)
I20260128 15:02:57.907759 20397 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:41873
I20260128 15:02:57.908877 20397 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:02:57.911688 20465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:57.917302 20465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b: Bootstrap starting.
I20260128 15:02:57.917780 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 20397
I20260128 15:02:57.917912 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:02:57.918147 13666 external_mini_cluster.cc:1468] Setting key b2dcee87c7db81dd4fa88b5b4827f22b
I20260128 15:02:57.919850 20465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:57.920723 20465 log.cc:826] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:57.922688 20465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b: No bootstrap required, opened a new log
Jan 28 15:02:57 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM
I20260128 15:02:57.925554 20465 raft_consensus.cc:359] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } }
I20260128 15:02:57.925766 20465 raft_consensus.cc:385] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:57.925833 20465 raft_consensus.cc:740] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 514b33d5af9e48e49897a8adce27696b, State: Initialized, Role: FOLLOWER
I20260128 15:02:57.926319 20465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } }
I20260128 15:02:57.926472 20465 raft_consensus.cc:399] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:02:57.926544 20465 raft_consensus.cc:493] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:02:57.926633 20465 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:57.927708 20465 raft_consensus.cc:515] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } }
I20260128 15:02:57.928104 20465 leader_election.cc:304] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 514b33d5af9e48e49897a8adce27696b; no voters:
I20260128 15:02:57.928399 20465 leader_election.cc:290] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:02:57.928543 20470 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:02:57.928731 20470 raft_consensus.cc:697] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [term 1 LEADER]: Becoming Leader. State: Replica: 514b33d5af9e48e49897a8adce27696b, State: Running, Role: LEADER
I20260128 15:02:57.929029 20470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } }
I20260128 15:02:57.929543 20465 sys_catalog.cc:565] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:57.930732 20472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "514b33d5af9e48e49897a8adce27696b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } }
I20260128 15:02:57.930867 20472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:57.931339 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:57.919393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56070 (local address 127.13.88.190:41873)
0128 15:02:57.919806 (+ 413us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:57.919818 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:57.919860 (+ 42us) server_negotiation.cc:408] Connection header received
0128 15:02:57.920600 (+ 740us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:57.920628 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:57.921010 (+ 382us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:57.921331 (+ 321us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:57.922162 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:57.923267 (+ 1105us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:57.923938 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:57.924254 (+ 316us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:57.926137 (+ 1883us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:57.926162 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:57.926175 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:57.926217 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:57.928134 (+ 1917us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:57.928708 (+ 574us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:57.928714 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:57.928720 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:57.928868 (+ 148us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:57.929161 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:57.929164 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:57.929166 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:57.929579 (+ 413us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:57.929793 (+ 214us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:57.930217 (+ 424us) server_negotiation.cc:300] Negotiation successful
0128 15:02:57.930487 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":120,"threads_started":1}
I20260128 15:02:57.931721 20471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 514b33d5af9e48e49897a8adce27696b. Latest consensus state: current_term: 1 leader_uuid: "514b33d5af9e48e49897a8adce27696b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } }
I20260128 15:02:57.931797 20479 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:02:57.931810 20471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:57.934604 20479 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:02:57.940091 20479 catalog_manager.cc:1357] Generated new cluster ID: 160ee1f718a440ada4dbc6a39dc86fae
I20260128 15:02:57.940171 20479 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:02:57.964571 20479 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:02:57.965533 20479 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:02:57.972448 20479 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b: Generated new TSK 0
I20260128 15:02:57.973140 20479 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
I20260128 15:02:58.023496 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:41873
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:58.151557 20493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:58.151803 20493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:58.151868 20493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:58.155280 20493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:58.155352 20493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:58.155457 20493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:02:58.159677 20493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:58.160727 20493 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:58.161554 20493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:58.167941 20501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:58.167943 20499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:58.167943 20498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:58.168365 20493 server_base.cc:1060] running on GCE node
I20260128 15:02:58.168752 20493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:58.169307 20493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:58.170452 20493 hybrid_clock.cc:648] HybridClock initialized: now 1769612578170438 us; error 27 us; skew 500 ppm
Jan 28 15:02:58 dist-test-slave-m28t krb5kdc[20381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612578, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:58.173009 20493 init.cc:377] Logged in from keytab as oryx/127.13.88.129@KRBTEST.COM (short username oryx)
I20260128 15:02:58.174103 20493 webserver.cc:492] Webserver started at http://127.13.88.129:43729/ using document root <none> and password file <none>
I20260128 15:02:58.174643 20493 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:58.174695 20493 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:58.174880 20493 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:58.176687 20493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "3fc2239548eb481580722ea108a90e21"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "fdf8de574eb2e3d1686576954215ef2f"
server_key_iv: "7ad0f8691be46644d2cc6dbef43ca769"
server_key_version: "encryptionkey@0"
I20260128 15:02:58.177171 20493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "3fc2239548eb481580722ea108a90e21"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "fdf8de574eb2e3d1686576954215ef2f"
server_key_iv: "7ad0f8691be46644d2cc6dbef43ca769"
server_key_version: "encryptionkey@0"
I20260128 15:02:58.180543 20493 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260128 15:02:58.182812 20508 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:58.183894 20493 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:58.184019 20493 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "3fc2239548eb481580722ea108a90e21"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "fdf8de574eb2e3d1686576954215ef2f"
server_key_iv: "7ad0f8691be46644d2cc6dbef43ca769"
server_key_version: "encryptionkey@0"
I20260128 15:02:58.184132 20493 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:58.223024 20493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:58.225948 20493 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:58.226137 20493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:58.226716 20493 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:58.227669 20493 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:58.227720 20493 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:58.227774 20493 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:58.227802 20493 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:58.236719 20493 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:33663
I20260128 15:02:58.236737 20621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:33663 every 8 connection(s)
I20260128 15:02:58.237716 20493 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:02:58.240409 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 20493
I20260128 15:02:58.240572 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:02:58.240851 13666 external_mini_cluster.cc:1468] Setting key d7d2f47d6498c9fb424f5cbf683fc505
Jan 28 15:02:58 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612578, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.13.88.130@KRBTEST.COM; defaulting to no policy
I20260128 15:02:58.251296 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.239568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:38913 (local address 127.13.88.190:41873)
0128 15:02:58.239726 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:58.239731 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:58.240577 (+ 846us) server_negotiation.cc:408] Connection header received
0128 15:02:58.241668 (+ 1091us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:58.241672 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:58.241720 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:58.241809 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:58.243937 (+ 2128us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.244540 (+ 603us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:58.245230 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.245435 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:58.247681 (+ 2246us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:58.247703 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:58.247706 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:58.247735 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:58.249114 (+ 1379us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:58.249754 (+ 640us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:58.249758 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:58.249759 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:58.249814 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:58.250148 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:58.250151 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:58.250152 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:58.250341 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:58.250427 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:58.251024 (+ 597us) server_negotiation.cc:300] Negotiation successful
0128 15:02:58.251154 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260128 15:02:58.252020 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.239867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.129:38913)
0128 15:02:58.240385 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:58.240470 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:58.241433 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:58.241945 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:58.241952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:58.242481 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:58.243743 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:58.243762 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.244684 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:58.244689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:58.245114 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:58.245121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.245306 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:58.245877 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:58.245906 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:58.247514 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:58.249244 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:58.249250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:58.249264 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:58.249627 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:58.249915 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:58.249919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:58.249921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:58.250058 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:58.250449 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:58.250455 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:58.250709 (+ 254us) client_negotiation.cc:769] Sending connection context
0128 15:02:58.250941 (+ 232us) client_negotiation.cc:240] Negotiation successful
0128 15:02:58.251183 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.88.130@KRBTEST.COM" created.
I20260128 15:02:58.253297 20622 heartbeater.cc:344] Connected to a master server at 127.13.88.190:41873
I20260128 15:02:58.253551 20622 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:58.254096 20622 heartbeater.cc:507] Master 127.13.88.190:41873 requested a full tablet report, sending...
I20260128 15:02:58.255625 20429 ts_manager.cc:194] Registered new tserver with Master: 3fc2239548eb481580722ea108a90e21 (127.13.88.129:33663)
I20260128 15:02:58.256896 20429 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.88.129@KRBTEST.COM'} at 127.13.88.129:38913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
I20260128 15:02:58.289510 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:41873
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:58.397897 20629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:58.398144 20629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:58.398206 20629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:58.401556 20629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:58.401628 20629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:58.401738 20629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:02:58.406018 20629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:58.407087 20629 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:58.407917 20629 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:58.414523 20634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:58.414561 20629 server_base.cc:1060] running on GCE node
W20260128 15:02:58.414510 20637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:58.414511 20635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:58.415079 20629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:58.415629 20629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:58.416769 20629 hybrid_clock.cc:648] HybridClock initialized: now 1769612578416752 us; error 30 us; skew 500 ppm
Jan 28 15:02:58 dist-test-slave-m28t krb5kdc[20381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612578, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:58.419250 20629 init.cc:377] Logged in from keytab as oryx/127.13.88.130@KRBTEST.COM (short username oryx)
I20260128 15:02:58.420271 20629 webserver.cc:492] Webserver started at http://127.13.88.130:34791/ using document root <none> and password file <none>
I20260128 15:02:58.420957 20629 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:58.421026 20629 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:58.421214 20629 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:58.422947 20629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "23361dc3c81bafe02befe28c1219e49b"
server_key_iv: "35d1c3eb0e15dd426b6a483efdd51952"
server_key_version: "encryptionkey@0"
I20260128 15:02:58.423422 20629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "23361dc3c81bafe02befe28c1219e49b"
server_key_iv: "35d1c3eb0e15dd426b6a483efdd51952"
server_key_version: "encryptionkey@0"
I20260128 15:02:58.426713 20629 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260128 15:02:58.428865 20644 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:58.429916 20629 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:58.430022 20629 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "23361dc3c81bafe02befe28c1219e49b"
server_key_iv: "35d1c3eb0e15dd426b6a483efdd51952"
server_key_version: "encryptionkey@0"
I20260128 15:02:58.430110 20629 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:58.466917 20629 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:58.469887 20629 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:58.470067 20629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:58.470634 20629 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:58.471539 20629 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:58.471588 20629 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:58.471640 20629 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:58.471668 20629 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:58.480365 20629 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:41581
I20260128 15:02:58.480389 20757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:41581 every 8 connection(s)
I20260128 15:02:58.481314 20629 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
I20260128 15:02:58.485512 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 20629
I20260128 15:02:58.485627 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:02:58.485865 13666 external_mini_cluster.cc:1468] Setting key 091c37e9e23185ca01c5c8a63833ceb1
Jan 28 15:02:58 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612578, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM
I20260128 15:02:58.493628 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.482987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:44801 (local address 127.13.88.190:41873)
0128 15:02:58.483130 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:58.483134 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:58.483788 (+ 654us) server_negotiation.cc:408] Connection header received
0128 15:02:58.484648 (+ 860us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:58.484652 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:58.484712 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:58.484782 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:58.486712 (+ 1930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.487353 (+ 641us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:58.487997 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.488211 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:58.490477 (+ 2266us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:58.490493 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:58.490496 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:58.490523 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:58.491812 (+ 1289us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:58.492264 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:58.492268 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:58.492269 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:58.492316 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:58.492597 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:58.492605 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:58.492607 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:58.492760 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:58.492843 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:58.493337 (+ 494us) server_negotiation.cc:300] Negotiation successful
0128 15:02:58.493471 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for oryx/127.13.88.131@KRBTEST.COM; defaulting to no policy
I20260128 15:02:58.494256 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.483233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.130:44801)
0128 15:02:58.483659 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:58.483695 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:58.484459 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:58.484945 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:58.484954 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:58.485371 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:58.486576 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:58.486594 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.487488 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:58.487491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:58.487889 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:58.487896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.488083 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:58.488687 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:58.488713 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:58.490326 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:58.491924 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:58.491930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:58.491943 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:58.492173 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:58.492393 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:58.492396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:58.492398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:58.492514 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:58.492855 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:58.492862 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:58.493097 (+ 235us) client_negotiation.cc:769] Sending connection context
0128 15:02:58.493270 (+ 173us) client_negotiation.cc:240] Negotiation successful
0128 15:02:58.493538 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":96,"threads_started":1}
I20260128 15:02:58.495339 20758 heartbeater.cc:344] Connected to a master server at 127.13.88.190:41873
I20260128 15:02:58.495579 20758 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.88.131@KRBTEST.COM" created.
I20260128 15:02:58.496134 20758 heartbeater.cc:507] Master 127.13.88.190:41873 requested a full tablet report, sending...
I20260128 15:02:58.497176 20429 ts_manager.cc:194] Registered new tserver with Master: e5e25f3cc9ae44aba196ed92a3adfb0e (127.13.88.130:41581)
I20260128 15:02:58.497884 20429 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.88.130@KRBTEST.COM'} at 127.13.88.130:44801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
I20260128 15:02:58.529107 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:41873
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:58.636289 20765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:58.636548 20765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:58.636617 20765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:58.639953 20765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:58.640043 20765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:58.640141 20765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:02:58.644634 20765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:58.645792 20765 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:58.646714 20765 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:58.653479 20770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:58.653483 20773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:58.653576 20765 server_base.cc:1060] running on GCE node
W20260128 15:02:58.653483 20771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:58.654084 20765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:58.654647 20765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:58.655812 20765 hybrid_clock.cc:648] HybridClock initialized: now 1769612578655795 us; error 30 us; skew 500 ppm
Jan 28 15:02:58 dist-test-slave-m28t krb5kdc[20381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612578, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:58.658486 20765 init.cc:377] Logged in from keytab as oryx/127.13.88.131@KRBTEST.COM (short username oryx)
I20260128 15:02:58.659600 20765 webserver.cc:492] Webserver started at http://127.13.88.131:34479/ using document root <none> and password file <none>
I20260128 15:02:58.660137 20765 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:58.660187 20765 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:58.660360 20765 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:58.662147 20765 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "fc94066dcfc943bdb867b2d77a4b8efe"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "f294204dbe9f77c29e162123250154c0"
server_key_iv: "2aad845b945a96eb5accbbf8ab2ee310"
server_key_version: "encryptionkey@0"
I20260128 15:02:58.662624 20765 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "fc94066dcfc943bdb867b2d77a4b8efe"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "f294204dbe9f77c29e162123250154c0"
server_key_iv: "2aad845b945a96eb5accbbf8ab2ee310"
server_key_version: "encryptionkey@0"
I20260128 15:02:58.666102 20765 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260128 15:02:58.668502 20780 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:58.669538 20765 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:58.669648 20765 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "fc94066dcfc943bdb867b2d77a4b8efe"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "f294204dbe9f77c29e162123250154c0"
server_key_iv: "2aad845b945a96eb5accbbf8ab2ee310"
server_key_version: "encryptionkey@0"
I20260128 15:02:58.669739 20765 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:58.687830 20765 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:58.690490 20765 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:58.690676 20765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:58.691221 20765 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:02:58.692098 20765 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:02:58.692145 20765 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:58.692198 20765 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:02:58.692226 20765 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:58.700274 20765 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:45859
I20260128 15:02:58.700300 20893 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:45859 every 8 connection(s)
I20260128 15:02:58.701295 20765 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
I20260128 15:02:58.704702 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 20765
I20260128 15:02:58.704811 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:02:58.705017 13666 external_mini_cluster.cc:1468] Setting key d8be0a6794b55de8b43c0b090f2b7eea
Jan 28 15:02:58 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612578, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM
I20260128 15:02:58.713035 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.702951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:46717 (local address 127.13.88.190:41873)
0128 15:02:58.703091 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:58.703095 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:58.703746 (+ 651us) server_negotiation.cc:408] Connection header received
0128 15:02:58.704642 (+ 896us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:58.704645 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:58.704709 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:58.704782 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:58.706691 (+ 1909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.707285 (+ 594us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:58.707901 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.708101 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:58.710285 (+ 2184us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:58.710304 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:58.710306 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:58.710337 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:58.711434 (+ 1097us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:58.711847 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:58.711849 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:58.711851 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:58.711887 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:58.712083 (+ 196us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:58.712085 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:58.712086 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:58.712230 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:58.712297 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:58.712800 (+ 503us) server_negotiation.cc:300] Negotiation successful
0128 15:02:58.712907 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260128 15:02:58.713697 20896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.703196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.131:46717)
0128 15:02:58.703607 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:58.703643 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:58.704401 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:58.704992 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:58.705001 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:58.705477 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:58.706540 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:58.706557 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.707405 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:58.707408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:58.707792 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:58.707799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.708019 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:58.708633 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:58.708661 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:58.710126 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:58.711534 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:58.711539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:58.711549 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:58.711774 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:58.711944 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:58.711946 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:58.711948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:58.712026 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:58.712295 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:58.712300 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:58.712587 (+ 287us) client_negotiation.cc:769] Sending connection context
0128 15:02:58.712771 (+ 184us) client_negotiation.cc:240] Negotiation successful
0128 15:02:58.712995 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
I20260128 15:02:58.714769 20894 heartbeater.cc:344] Connected to a master server at 127.13.88.190:41873
I20260128 15:02:58.715022 20894 heartbeater.cc:461] Registering TS with master...
I20260128 15:02:58.715583 20894 heartbeater.cc:507] Master 127.13.88.190:41873 requested a full tablet report, sending...
I20260128 15:02:58.716719 20429 ts_manager.cc:194] Registered new tserver with Master: fc94066dcfc943bdb867b2d77a4b8efe (127.13.88.131:45859)
I20260128 15:02:58.717372 20429 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.88.131@KRBTEST.COM'} at 127.13.88.131:46717
I20260128 15:02:58.718716 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.88.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.88.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.88.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.88.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.88.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.88.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
I20260128 15:02:58.765226 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.189:38211
--webserver_interface=127.13.88.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:41873,127.13.88.189:38211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:02:58.872632 20901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:58.872902 20901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:58.872974 20901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:58.876535 20901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:58.876614 20901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:58.876643 20901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:58.876669 20901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:58.876700 20901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:02:58.881132 20901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:41873,127.13.88.189:38211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.189:38211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.88.189
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
W20260128 15:02:58.881364 20901 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.13.88.190:41873,127.13.88.189:38211'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260128 15:02:58.882275 20901 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:58.883136 20901 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:58.888660 20906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:58.888657 20907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:58.888655 20909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:58.888959 20901 server_base.cc:1060] running on GCE node
I20260128 15:02:58.889559 20901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:58.890544 20901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:58.891690 20901 hybrid_clock.cc:648] HybridClock initialized: now 1769612578891674 us; error 30 us; skew 500 ppm
Jan 28 15:02:58 dist-test-slave-m28t krb5kdc[20381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612578, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:58.894598 20901 init.cc:377] Logged in from keytab as oryx/127.13.88.189@KRBTEST.COM (short username oryx)
I20260128 15:02:58.895758 20901 webserver.cc:492] Webserver started at http://127.13.88.189:46599/ using document root <none> and password file <none>
I20260128 15:02:58.896350 20901 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:58.896442 20901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:58.896652 20901 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:58.898429 20901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/data/instance:
uuid: "1c36b3bafb1547efb54a35126ed2c62c"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "2919a47ae876eeb0cadcdc6c91584d58"
server_key_iv: "ff5f1ffefa6d3e63ec6054aff0d58d67"
server_key_version: "encryptionkey@0"
I20260128 15:02:58.898892 20901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/wal/instance:
uuid: "1c36b3bafb1547efb54a35126ed2c62c"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "2919a47ae876eeb0cadcdc6c91584d58"
server_key_iv: "ff5f1ffefa6d3e63ec6054aff0d58d67"
server_key_version: "encryptionkey@0"
I20260128 15:02:58.902341 20901 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260128 15:02:58.904497 20916 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:58.905503 20901 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:58.905606 20901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/wal
uuid: "1c36b3bafb1547efb54a35126ed2c62c"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "2919a47ae876eeb0cadcdc6c91584d58"
server_key_iv: "ff5f1ffefa6d3e63ec6054aff0d58d67"
server_key_version: "encryptionkey@0"
I20260128 15:02:58.905699 20901 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:58.917974 20901 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:58.920527 20901 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:58.920694 20901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:58.927430 20968 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.189:38211 every 8 connection(s)
I20260128 15:02:58.927439 20901 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.189:38211
I20260128 15:02:58.928498 20901 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/data/info.pb
I20260128 15:02:58.930785 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 20901
I20260128 15:02:58.931311 20969 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:58.935331 20969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } has no permanent_uuid. Determining permanent_uuid...
Jan 28 15:02:58 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.88.189@KRBTEST.COM
Jan 28 15:02:58 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612578, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.189@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM
I20260128 15:02:58.943257 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.932053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.13.88.189:38211)
0128 15:02:58.932487 (+ 434us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:58.932496 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:58.932529 (+ 33us) server_negotiation.cc:408] Connection header received
0128 15:02:58.933571 (+ 1042us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:58.933590 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:58.933871 (+ 281us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:58.934165 (+ 294us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:58.935052 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.935837 (+ 785us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:58.936514 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.937045 (+ 531us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:58.938935 (+ 1890us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:58.938984 (+ 49us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:58.938993 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:58.939028 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:58.940820 (+ 1792us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:58.941223 (+ 403us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:58.941228 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:58.941231 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:58.941298 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:58.941508 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:58.941511 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:58.941513 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:58.941850 (+ 337us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:58.942030 (+ 180us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:58.942314 (+ 284us) server_negotiation.cc:300] Negotiation successful
0128 15:02:58.942559 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":123,"threads_started":1}
I20260128 15:02:58.944911 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.936850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.13.88.190:41873)
0128 15:02:58.936989 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:58.936993 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:58.938139 (+ 1146us) server_negotiation.cc:408] Connection header received
0128 15:02:58.938237 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:58.938240 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:58.938296 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:58.938387 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:58.939230 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.939709 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:58.940283 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.940507 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:58.942207 (+ 1700us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:58.942224 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:58.942226 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:58.942250 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:58.943463 (+ 1213us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:58.943856 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:58.943860 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:58.943861 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:58.943905 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:58.944139 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:58.944143 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:58.944145 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:58.944308 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:58.944392 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:58.944707 (+ 315us) server_negotiation.cc:300] Negotiation successful
0128 15:02:58.944799 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260128 15:02:58.944967 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.936764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:56074)
0128 15:02:58.938026 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:58.938058 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:58.938154 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:58.938433 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:58.938437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:58.938821 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:58.939109 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:58.939117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.939811 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:58.939814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:58.940185 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:58.940192 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.940287 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:58.940752 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:58.940764 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:58.942094 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:58.943559 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:58.943567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:58.943572 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:58.943770 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:58.943979 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:58.943983 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:58.943984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:58.944034 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:58.944408 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:58.944437 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:58.944540 (+ 103us) client_negotiation.cc:769] Sending connection context
0128 15:02:58.944670 (+ 130us) client_negotiation.cc:240] Negotiation successful
0128 15:02:58.944804 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"thread_start_us":124,"threads_started":1}
I20260128 15:02:58.946033 20969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } has no permanent_uuid. Determining permanent_uuid...
Jan 28 15:02:58 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612578, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.189@KRBTEST.COM for oryx/127.13.88.189@KRBTEST.COM
I20260128 15:02:58.951977 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.946486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.13.88.189:38211)
0128 15:02:58.946577 (+ 91us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:58.946581 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:58.946594 (+ 13us) server_negotiation.cc:408] Connection header received
0128 15:02:58.946696 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:58.946699 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:58.946732 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:58.946787 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:58.947193 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.947629 (+ 436us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:58.948118 (+ 489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.948285 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:58.949652 (+ 1367us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:58.949665 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:58.949667 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:58.949686 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:58.950778 (+ 1092us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:58.951094 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:58.951097 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:58.951099 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:58.951135 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:58.951319 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:58.951331 (+ 12us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:58.951333 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:58.951451 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:58.951530 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:58.951731 (+ 201us) server_negotiation.cc:300] Negotiation successful
0128 15:02:58.951858 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":37}
I20260128 15:02:58.951977 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.946447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.189:38211 (local address 127.0.0.1:57680)
0128 15:02:58.946548 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:58.946561 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:58.946627 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:58.946806 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:58.946809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:58.946936 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:58.947100 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:58.947105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.947715 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:58.947718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:58.948033 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:58.948038 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.948133 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:58.948535 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:58.948547 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:58.949554 (+ 1007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:58.950874 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:58.950877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:58.950880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:58.951018 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:58.951203 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:58.951206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:58.951208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:58.951252 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:58.951530 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:58.951534 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:58.951602 (+ 68us) client_negotiation.cc:769] Sending connection context
0128 15:02:58.951683 (+ 81us) client_negotiation.cc:240] Negotiation successful
0128 15:02:58.951807 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260128 15:02:58.954743 20969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: Bootstrap starting.
I20260128 15:02:58.957302 20969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:58.958029 20969 log.cc:826] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:58.959728 20969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: No bootstrap required, opened a new log
I20260128 15:02:58.962210 20969 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } }
I20260128 15:02:58.962378 20969 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:58.962432 20969 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c36b3bafb1547efb54a35126ed2c62c, State: Initialized, Role: FOLLOWER
I20260128 15:02:58.962838 20969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } }
I20260128 15:02:58.963249 20975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } } }
I20260128 15:02:58.963356 20975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: This master's current role is: FOLLOWER
I20260128 15:02:58.963848 20969 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: configured and running, proceeding with master startup.
W20260128 15:02:58.969285 20986 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 15:02:58.969384 20986 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 15:02:58.970264 20901 master_runner.cc:428] Detected that this master 1c36b3bafb1547efb54a35126ed2c62c is joining an existing cluster
I20260128 15:02:58.970333 20901 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.88.189:38211
I20260128 15:02:58.979315 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.973151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57696 (local address 127.13.88.189:38211)
0128 15:02:58.973248 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:58.973252 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:58.973422 (+ 170us) server_negotiation.cc:408] Connection header received
0128 15:02:58.973527 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:58.973530 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:58.973566 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:58.973616 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:58.974246 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.974789 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:58.975563 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.975733 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:58.976595 (+ 862us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:58.976608 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:58.976610 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:58.976630 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:58.977911 (+ 1281us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:58.978343 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:58.978346 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:58.978347 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:58.978389 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:58.978600 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:58.978602 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:58.978603 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:58.978725 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:58.978790 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:58.979029 (+ 239us) server_negotiation.cc:300] Negotiation successful
0128 15:02:58.979186 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":38}
I20260128 15:02:58.979341 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.973103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.189:38211 (local address 127.0.0.1:57696)
0128 15:02:58.973350 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:58.973360 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:58.973447 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:58.973722 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:58.973725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:58.973925 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:58.974143 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:58.974149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.974897 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:58.974901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:58.975461 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:58.975470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.975579 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:58.976065 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:58.976081 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:58.976492 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:58.978011 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:58.978015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:58.978017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:58.978227 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:58.978471 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:58.978475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:58.978477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:58.978530 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:58.978799 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:58.978802 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:58.978887 (+ 85us) client_negotiation.cc:769] Sending connection context
0128 15:02:58.978990 (+ 103us) client_negotiation.cc:240] Negotiation successful
0128 15:02:58.979121 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260128 15:02:58.992549 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.972732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:56088)
0128 15:02:58.973007 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:58.973027 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:58.973170 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:58.973548 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:58.973551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:58.973849 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:58.974482 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:58.974492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.975313 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:58.975317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:58.975772 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:58.975780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.975896 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:58.989358 (+ 13462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:58.989374 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:58.989670 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:58.991233 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:58.991237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:58.991239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:58.991396 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:58.991624 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:58.991627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:58.991628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:58.991671 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:58.991998 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:58.992001 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:58.992087 (+ 86us) client_negotiation.cc:769] Sending connection context
0128 15:02:58.992184 (+ 97us) client_negotiation.cc:240] Negotiation successful
0128 15:02:58.992328 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":90,"threads_started":1}
I20260128 15:02:58.992542 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.972800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.13.88.190:41873)
0128 15:02:58.972941 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:58.972946 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:58.973099 (+ 153us) server_negotiation.cc:408] Connection header received
0128 15:02:58.973288 (+ 189us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:58.973292 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:58.973353 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:58.973405 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:58.974601 (+ 1196us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.975204 (+ 603us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:58.975873 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.976052 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:58.989840 (+ 13788us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:58.989862 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:58.989864 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:58.989895 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:58.991113 (+ 1218us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:58.991490 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:58.991493 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:58.991495 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:58.991544 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:58.991748 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:58.991751 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:58.991753 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:58.991914 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:58.991997 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:58.992266 (+ 269us) server_negotiation.cc:300] Negotiation successful
0128 15:02:58.992380 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jan 28 15:02:59 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612577, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.190@KRBTEST.COM for oryx/127.13.88.189@KRBTEST.COM
WARNING: no policy specified for oryx/127.13.88.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.88.188@KRBTEST.COM" created.
I20260128 15:02:59.006474 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.997072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.13.88.189:38211)
0128 15:02:58.997198 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:58.997202 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:58.997572 (+ 370us) server_negotiation.cc:408] Connection header received
0128 15:02:58.997829 (+ 257us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:58.997833 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:58.997884 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:58.997931 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:58.999046 (+ 1115us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.999713 (+ 667us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.000537 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.000730 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.003022 (+ 2292us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:59.003044 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:59.003047 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:59.003077 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:59.004791 (+ 1714us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.005223 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.005227 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.005229 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.005273 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.005513 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.005516 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.005518 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.005724 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:59.005822 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:59.006226 (+ 404us) server_negotiation.cc:300] Negotiation successful
0128 15:02:59.006324 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260128 15:02:59.006969 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:58.997023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.189:38211 (local address 127.0.0.1:57712)
0128 15:02:58.997464 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:58.997506 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:58.997693 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:58.998098 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:58.998103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:58.998607 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:58.998909 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:58.998918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:58.999825 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:58.999829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:59.000401 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.000411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.000547 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.001046 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:59.001063 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:59.002886 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:59.004896 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.004904 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.004908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.005140 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.005358 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.005361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.005363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.005418 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.005814 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:59.005820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:59.005974 (+ 154us) client_negotiation.cc:769] Sending connection context
0128 15:02:59.006632 (+ 658us) client_negotiation.cc:240] Negotiation successful
0128 15:02:59.006809 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":96,"threads_started":1}
I20260128 15:02:59.009228 20428 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.13.88.189:38211: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c"
member_type: NON_VOTER
last_known_addr {
host: "127.13.88.189"
port: 38211
}
attrs {
promote: true
}
}
dest_uuid: "514b33d5af9e48e49897a8adce27696b"
cas_config_opid_index: -1
I20260128 15:02:59.009922 20428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: NON_VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: true } }
I20260128 15:02:59.011955 20998 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1c36b3bafb1547efb54a35126ed2c62c (127.13.88.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: NON_VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: true } } }
I20260128 15:02:59.012638 20943 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:02:59.013607 20998 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.13.88.189:38211
I20260128 15:02:59.013921 20943 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 FOLLOWER]: Refusing update from remote peer 514b33d5af9e48e49897a8adce27696b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260128 15:02:59.014096 20901 master.cc:561] Master@127.13.88.189:38211 shutting down...
I20260128 15:02:59.013860 20998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "514b33d5af9e48e49897a8adce27696b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: NON_VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: true } } }
I20260128 15:02:59.014317 20998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:59.014554 20998 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: NON_VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260128 15:02:59.016341 20416 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260128 15:02:59.017341 20416 consensus_peers.cc:597] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b -> Peer 1c36b3bafb1547efb54a35126ed2c62c (127.13.88.189:38211): Couldn't send request to peer 1c36b3bafb1547efb54a35126ed2c62c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260128 15:02:59.018419 20901 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:02:59.018662 20901 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:59.018746 20901 tablet_replica.cc:333] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.88.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.88.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.88.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.188@KRBTEST.COM" created.
I20260128 15:02:59.032989 20901 master.cc:583] Master@127.13.88.189:38211 shutdown complete.
I20260128 15:02:59.033051 20901 master_runner.cc:305] Clearing existing system tablet
I20260128 15:02:59.033288 20901 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:59.033340 20901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:59.034498 20901 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260128 15:02:59.035285 21005 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:59.035501 20901 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:59.035620 20901 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260128 15:02:59.035655 20901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/wal
uuid: "1c36b3bafb1547efb54a35126ed2c62c"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "2919a47ae876eeb0cadcdc6c91584d58"
server_key_iv: "ff5f1ffefa6d3e63ec6054aff0d58d67"
server_key_version: "encryptionkey@0"
I20260128 15:02:59.037284 20901 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:59.038975 20901 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260128 15:02:59.039050 20901 log.cc:1199] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260128 15:02:59.039259 20901 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: Deleting consensus metadata
I20260128 15:02:59.039448 20901 master_runner.cc:315] Copying system tablet from 127.13.88.190:41873
I20260128 15:02:59.040552 20901 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: tablet copy: Beginning tablet copy session from remote peer at address 127.13.88.190:41873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.88.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab.
I20260128 15:02:59.048240 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.188:44165
--webserver_interface=127.13.88.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:41873,127.13.88.189:38211,127.13.88.188:44165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260128 15:02:59.049500 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.041224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56104 (local address 127.13.88.190:41873)
0128 15:02:59.041353 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:59.041358 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:59.041412 (+ 54us) server_negotiation.cc:408] Connection header received
0128 15:02:59.041574 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:59.041577 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:59.041637 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:59.041714 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:59.042386 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.042906 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.043745 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.043887 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.044820 (+ 933us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:59.044842 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:59.044845 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:59.044872 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:59.047831 (+ 2959us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.048308 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.048312 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.048315 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.048373 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.048701 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.048704 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.048706 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.048888 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:59.048981 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:59.049227 (+ 246us) server_negotiation.cc:300] Negotiation successful
0128 15:02:59.049345 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260128 15:02:59.049511 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.041039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:56104)
0128 15:02:59.041334 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:59.041350 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:59.041467 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:59.041725 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:59.041729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:59.041924 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:59.042218 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.042227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.043014 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.043018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:59.043634 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.043643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.043759 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.044323 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:59.044344 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:59.044696 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:59.047974 (+ 3278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.047978 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.047981 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.048205 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.048541 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.048545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.048547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.048616 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.048976 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:59.048979 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:59.049073 (+ 94us) client_negotiation.cc:769] Sending connection context
0128 15:02:59.049174 (+ 101us) client_negotiation.cc:240] Negotiation successful
0128 15:02:59.049323 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":88,"threads_started":1}
I20260128 15:02:59.050115 20449 tablet_copy_service.cc:140] P 514b33d5af9e48e49897a8adce27696b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1c36b3bafb1547efb54a35126ed2c62c ({username='oryx', principal='oryx/127.13.88.189@KRBTEST.COM'} at 127.0.0.1:56104)
I20260128 15:02:59.050280 20449 tablet_copy_service.cc:161] P 514b33d5af9e48e49897a8adce27696b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1c36b3bafb1547efb54a35126ed2c62c at {username='oryx', principal='oryx/127.13.88.189@KRBTEST.COM'} at 127.0.0.1:56104: session id = 1c36b3bafb1547efb54a35126ed2c62c-00000000000000000000000000000000
I20260128 15:02:59.051923 20449 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b: Tablet Copy: opened 0 blocks and 1 log segments
I20260128 15:02:59.053891 20901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260128 15:02:59.056234 20901 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: tablet copy: Starting download of 0 data blocks...
I20260128 15:02:59.056468 20901 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: tablet copy: Starting download of 1 WAL segments...
I20260128 15:02:59.058733 20901 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260128 15:02:59.060640 20449 tablet_copy_service.cc:342] P 514b33d5af9e48e49897a8adce27696b: Request end of tablet copy session 1c36b3bafb1547efb54a35126ed2c62c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.88.189@KRBTEST.COM'} at 127.0.0.1:56104
I20260128 15:02:59.060776 20449 tablet_copy_service.cc:434] P 514b33d5af9e48e49897a8adce27696b: ending tablet copy session 1c36b3bafb1547efb54a35126ed2c62c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1c36b3bafb1547efb54a35126ed2c62c
W20260128 15:02:59.066120 20901 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260128 15:02:59.068475 20901 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:59.070201 21013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:59.073441 21014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:59.073658 20901 server_base.cc:1060] running on GCE node
W20260128 15:02:59.073658 21016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:59.073940 20901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:59.074322 20901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:59.075546 20901 hybrid_clock.cc:648] HybridClock initialized: now 1769612579075530 us; error 36 us; skew 500 ppm
Jan 28 15:02:59 dist-test-slave-m28t krb5kdc[20381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612579, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:59.077644 20901 init.cc:377] Logged in from keytab as oryx/127.13.88.189@KRBTEST.COM (short username oryx)
I20260128 15:02:59.078218 20901 webserver.cc:492] Webserver started at http://127.13.88.189:40467/ using document root <none> and password file <none>
I20260128 15:02:59.078459 20901 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:59.078547 20901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:59.079871 20901 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:59.080865 21023 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:59.081111 20901 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:59.081180 20901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/wal
uuid: "1c36b3bafb1547efb54a35126ed2c62c"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "2919a47ae876eeb0cadcdc6c91584d58"
server_key_iv: "ff5f1ffefa6d3e63ec6054aff0d58d67"
server_key_version: "encryptionkey@0"
I20260128 15:02:59.081250 20901 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:59.091140 20901 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:59.093227 20901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:59.097790 20901 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.189:38211
I20260128 15:02:59.097886 21075 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.189:38211 every 8 connection(s)
I20260128 15:02:59.098615 20901 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-1/data/info.pb
I20260128 15:02:59.100513 21076 sys_catalog.cc:263] Verifying existing consensus state
I20260128 15:02:59.101447 21076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: Bootstrap starting.
I20260128 15:02:59.111778 21076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260128 15:02:59.112066 21076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: Bootstrap complete.
I20260128 15:02:59.112686 21076 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: NON_VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: true } }
I20260128 15:02:59.112818 21076 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c36b3bafb1547efb54a35126ed2c62c, State: Initialized, Role: LEARNER
I20260128 15:02:59.112926 21076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: NON_VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: true } }
I20260128 15:02:59.113229 21079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: NON_VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: true } } }
I20260128 15:02:59.113375 21076 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:59.113409 21079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: This master's current role is: LEARNER
I20260128 15:02:59.116632 21090 catalog_manager.cc:1269] Loaded cluster ID: 160ee1f718a440ada4dbc6a39dc86fae
I20260128 15:02:59.116689 21090 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: loading cluster ID for follower catalog manager: success
I20260128 15:02:59.118834 21090 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: acquiring CA information for follower catalog manager: success
I20260128 15:02:59.120184 21090 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260128 15:02:59.169932 21012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:02:59.170153 21012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:02:59.170220 21012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:02:59.173492 21012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:02:59.173559 21012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:02:59.173599 21012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:02:59.173631 21012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:02:59.173655 21012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:02:59.177961 21012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:41873,127.13.88.189:38211,127.13.88.188:44165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.188:44165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.88.188
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:02:59.178963 21012 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:02:59.179764 21012 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:59.185127 21096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:59.185144 21095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:59.185336 21012 server_base.cc:1060] running on GCE node
W20260128 15:02:59.185130 21098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:59.185901 21012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:59.186771 21012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:59.187920 21012 hybrid_clock.cc:648] HybridClock initialized: now 1769612579187908 us; error 25 us; skew 500 ppm
Jan 28 15:02:59 dist-test-slave-m28t krb5kdc[20381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612579, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:59.190478 21012 init.cc:377] Logged in from keytab as oryx/127.13.88.188@KRBTEST.COM (short username oryx)
I20260128 15:02:59.191529 21012 webserver.cc:492] Webserver started at http://127.13.88.188:39943/ using document root <none> and password file <none>
I20260128 15:02:59.192036 21012 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:59.192086 21012 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:59.192246 21012 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:02:59.193917 21012 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/data/instance:
uuid: "c103d7f4e85045059b10b02c5788eaba"
format_stamp: "Formatted at 2026-01-28 15:02:59 on dist-test-slave-m28t"
server_key: "9ecb1ef899e3c58a280cd645964c1cbb"
server_key_iv: "e266457e0c1f97690234d50cf78c3ade"
server_key_version: "encryptionkey@0"
I20260128 15:02:59.194352 21012 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/wal/instance:
uuid: "c103d7f4e85045059b10b02c5788eaba"
format_stamp: "Formatted at 2026-01-28 15:02:59 on dist-test-slave-m28t"
server_key: "9ecb1ef899e3c58a280cd645964c1cbb"
server_key_iv: "e266457e0c1f97690234d50cf78c3ade"
server_key_version: "encryptionkey@0"
I20260128 15:02:59.197609 21012 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260128 15:02:59.199612 21105 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:59.200619 21012 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:02:59.200716 21012 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/wal
uuid: "c103d7f4e85045059b10b02c5788eaba"
format_stamp: "Formatted at 2026-01-28 15:02:59 on dist-test-slave-m28t"
server_key: "9ecb1ef899e3c58a280cd645964c1cbb"
server_key_iv: "e266457e0c1f97690234d50cf78c3ade"
server_key_version: "encryptionkey@0"
I20260128 15:02:59.200798 21012 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:59.220314 21012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:59.222887 21012 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:02:59.223058 21012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:59.229848 21157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.188:44165 every 8 connection(s)
I20260128 15:02:59.229858 21012 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.188:44165
I20260128 15:02:59.230867 21012 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/data/info.pb
I20260128 15:02:59.233554 21158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:02:59.234604 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 21012
I20260128 15:02:59.237579 21158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } has no permanent_uuid. Determining permanent_uuid...
Jan 28 15:02:59 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.88.188@KRBTEST.COM
Jan 28 15:02:59 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612579, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.188@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM
I20260128 15:02:59.247541 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.239236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56114 (local address 127.13.88.190:41873)
0128 15:02:59.239718 (+ 482us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:59.239721 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:59.239999 (+ 278us) server_negotiation.cc:408] Connection header received
0128 15:02:59.240183 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:59.240186 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:59.240256 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:59.240335 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:59.241283 (+ 948us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.241850 (+ 567us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.242506 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.242988 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.244638 (+ 1650us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:59.244658 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:59.244662 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:59.244693 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:59.246001 (+ 1308us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.246475 (+ 474us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.246479 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.246481 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.246530 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.246778 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.246782 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.246783 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.246930 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:59.247005 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:59.247345 (+ 340us) server_negotiation.cc:300] Negotiation successful
0128 15:02:59.247433 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395}
I20260128 15:02:59.247536 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.235856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34014 (local address 127.13.88.188:44165)
0128 15:02:59.236287 (+ 431us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:59.236296 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:59.236331 (+ 35us) server_negotiation.cc:408] Connection header received
0128 15:02:59.237007 (+ 676us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:59.237029 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:59.237398 (+ 369us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:59.237722 (+ 324us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:59.238691 (+ 969us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.239967 (+ 1276us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.240755 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.240966 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.243164 (+ 2198us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:59.243180 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:59.243188 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:59.243209 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:59.244946 (+ 1737us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.245330 (+ 384us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.245336 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.245341 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.245410 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.245612 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.245616 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.245617 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.246048 (+ 431us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:59.246210 (+ 162us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:59.246509 (+ 299us) server_negotiation.cc:300] Negotiation successful
0128 15:02:59.246759 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":136,"threads_started":1}
I20260128 15:02:59.247583 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.239102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:56114)
0128 15:02:59.239884 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:59.239917 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:59.240028 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:59.240393 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:59.240399 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:59.240902 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:59.241147 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.241155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.241955 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.241958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:59.242393 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.242401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.242512 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.242962 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:59.242985 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:59.244516 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:59.246110 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.246119 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.246123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.246374 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.246633 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.246636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.246638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.246689 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.247051 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:59.247057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:59.247163 (+ 106us) client_negotiation.cc:769] Sending connection context
0128 15:02:59.247303 (+ 140us) client_negotiation.cc:240] Negotiation successful
0128 15:02:59.247446 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":113,"threads_started":1}
I20260128 15:02:59.248711 21158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } has no permanent_uuid. Determining permanent_uuid...
Jan 28 15:02:59 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612579, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.188@KRBTEST.COM for oryx/127.13.88.189@KRBTEST.COM
I20260128 15:02:59.255674 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.249161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.189:38211 (local address 127.0.0.1:57716)
0128 15:02:59.249321 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:59.249335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:59.249480 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:59.249697 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:59.249699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:59.249828 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:59.250032 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.250038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.250887 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.250889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:59.251279 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.251286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.251369 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.251744 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:59.251756 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:59.252872 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:59.254517 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.254520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.254522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.254673 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.254889 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.254892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.254893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.254937 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.255250 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:59.255253 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:59.255331 (+ 78us) client_negotiation.cc:769] Sending connection context
0128 15:02:59.255413 (+ 82us) client_negotiation.cc:240] Negotiation successful
0128 15:02:59.255524 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260128 15:02:59.255720 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.249244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57716 (local address 127.13.88.189:38211)
0128 15:02:59.249522 (+ 278us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:59.249525 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:59.249537 (+ 12us) server_negotiation.cc:408] Connection header received
0128 15:02:59.249571 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:59.249573 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:59.249610 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:59.249681 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:59.250122 (+ 441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.250788 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.251362 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.251496 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.253001 (+ 1505us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:59.253022 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:59.253025 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:59.253054 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:59.254409 (+ 1355us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.254750 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.254752 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.254753 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.254809 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.255013 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.255016 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.255018 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.255166 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:59.255249 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:59.255471 (+ 222us) server_negotiation.cc:300] Negotiation successful
0128 15:02:59.255582 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260128 15:02:59.256385 21158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } has no permanent_uuid. Determining permanent_uuid...
Jan 28 15:02:59 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612579, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.188@KRBTEST.COM for oryx/127.13.88.188@KRBTEST.COM
I20260128 15:02:59.259801 20622 heartbeater.cc:499] Master 127.13.88.190:41873 was elected leader, sending a full tablet report...
I20260128 15:02:59.262596 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.256814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.188:44165 (local address 127.0.0.1:34028)
0128 15:02:59.256929 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:59.256941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:59.257022 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:59.257214 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:59.257217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:59.257339 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:59.257483 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.257488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.258078 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.258081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:59.258409 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.258415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.258493 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.258857 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:59.258868 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:59.259923 (+ 1055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:59.261548 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.261551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.261552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.261693 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.261853 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.261855 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.261856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.261891 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.262216 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:59.262218 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:59.262291 (+ 73us) client_negotiation.cc:769] Sending connection context
0128 15:02:59.262375 (+ 84us) client_negotiation.cc:240] Negotiation successful
0128 15:02:59.262469 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260128 15:02:59.262635 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.256862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34028 (local address 127.13.88.188:44165)
0128 15:02:59.256967 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:59.256970 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:59.256981 (+ 11us) server_negotiation.cc:408] Connection header received
0128 15:02:59.257094 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:59.257096 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:59.257130 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:59.257187 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:59.257564 (+ 377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.257994 (+ 430us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.258483 (+ 489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.258607 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.260026 (+ 1419us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:59.260042 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:59.260045 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:59.260085 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:59.261457 (+ 1372us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.261763 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.261766 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.261767 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.261800 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.261942 (+ 142us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.261944 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.261945 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.262109 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:59.262179 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:59.262419 (+ 240us) server_negotiation.cc:300] Negotiation successful
0128 15:02:59.262518 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":37}
I20260128 15:02:59.265225 21158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: Bootstrap starting.
I20260128 15:02:59.267535 21158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: Neither blocks nor log segments found. Creating new log.
I20260128 15:02:59.268216 21158 log.cc:826] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: Log is configured to *not* fsync() on all Append() calls
I20260128 15:02:59.269984 21158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: No bootstrap required, opened a new log
I20260128 15:02:59.272382 21158 raft_consensus.cc:359] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } }
I20260128 15:02:59.272586 21158 raft_consensus.cc:385] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:02:59.272644 21158 raft_consensus.cc:740] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c103d7f4e85045059b10b02c5788eaba, State: Initialized, Role: FOLLOWER
I20260128 15:02:59.273041 21158 consensus_queue.cc:260] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } }
I20260128 15:02:59.273448 21164 sys_catalog.cc:455] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } } }
I20260128 15:02:59.273551 21164 sys_catalog.cc:458] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [sys.catalog]: This master's current role is: FOLLOWER
I20260128 15:02:59.274108 21158 sys_catalog.cc:565] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [sys.catalog]: configured and running, proceeding with master startup.
W20260128 15:02:59.278276 21175 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 15:02:59.278342 21175 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 15:02:59.281486 21012 master_runner.cc:428] Detected that this master c103d7f4e85045059b10b02c5788eaba is joining an existing cluster
I20260128 15:02:59.281534 21012 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.88.188:44165
I20260128 15:02:59.304237 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.301741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:56136)
0128 15:02:59.301866 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:59.301880 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:59.301954 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:59.302322 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:59.302325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:59.302340 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:59.302518 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.302524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.303220 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.303224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:59.303794 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.303800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.303901 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.303969 (+ 68us) client_negotiation.cc:769] Sending connection context
0128 15:02:59.304022 (+ 53us) client_negotiation.cc:240] Negotiation successful
0128 15:02:59.304094 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"spinlock_wait_cycles":1920}
I20260128 15:02:59.304652 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.301809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.13.88.190:41873)
0128 15:02:59.302135 (+ 326us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:59.302139 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:59.302148 (+ 9us) server_negotiation.cc:408] Connection header received
0128 15:02:59.302189 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:59.302191 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:59.302229 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:59.302300 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:59.302613 (+ 313us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.303123 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.303897 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.304309 (+ 412us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.304387 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:59.304468 (+ 81us) server_negotiation.cc:300] Negotiation successful
0128 15:02:59.304540 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":135,"threads_started":1}
I20260128 15:02:59.307824 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.305648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.189:38211 (local address 127.0.0.1:57718)
0128 15:02:59.305754 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:59.305767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:59.305831 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:59.306116 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:59.306119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:59.306137 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:59.306277 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.306282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.306947 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.306950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:59.307500 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.307507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.307600 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.307616 (+ 16us) client_negotiation.cc:769] Sending connection context
0128 15:02:59.307661 (+ 45us) client_negotiation.cc:240] Negotiation successful
0128 15:02:59.307716 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260128 15:02:59.308357 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.305774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.13.88.189:38211)
0128 15:02:59.305918 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:59.305921 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:59.305932 (+ 11us) server_negotiation.cc:408] Connection header received
0128 15:02:59.305970 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:59.305972 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:59.306019 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:59.306095 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:59.306383 (+ 288us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.306843 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.307615 (+ 772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.308060 (+ 445us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.308167 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:59.308208 (+ 41us) server_negotiation.cc:300] Negotiation successful
0128 15:02:59.308251 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260128 15:02:59.317637 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.308830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.13.88.189:38211)
0128 15:02:59.308931 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:59.308935 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:59.309450 (+ 515us) server_negotiation.cc:408] Connection header received
0128 15:02:59.309643 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:59.309647 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:59.309690 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:59.309736 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:59.310428 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.311096 (+ 668us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.312171 (+ 1075us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.312348 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.313808 (+ 1460us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:59.313824 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:59.313828 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:59.313851 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:59.316084 (+ 2233us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.316505 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.316508 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.316510 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.316551 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.316789 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.316792 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.316794 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.316957 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:59.317025 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:59.317376 (+ 351us) server_negotiation.cc:300] Negotiation successful
0128 15:02:59.317493 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260128 15:02:59.317866 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.283500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34044 (local address 127.13.88.188:44165)
0128 15:02:59.283606 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:59.283611 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:59.283954 (+ 343us) server_negotiation.cc:408] Connection header received
0128 15:02:59.284094 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:59.284097 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:59.284133 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:59.284198 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:59.310116 (+ 25918us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.310801 (+ 685us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.312487 (+ 1686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.312662 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.313976 (+ 1314us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:59.313991 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:59.313994 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:59.314014 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:59.315724 (+ 1710us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.316882 (+ 1158us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.316885 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.316887 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.316923 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.317119 (+ 196us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.317122 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.317124 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.317280 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:59.317365 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:59.317618 (+ 253us) server_negotiation.cc:300] Negotiation successful
0128 15:02:59.317719 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260128 15:02:59.318073 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.309033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.189:38211 (local address 127.0.0.1:57728)
0128 15:02:59.309373 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:59.309387 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:59.309541 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:59.309850 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:59.309853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:59.310046 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:59.310299 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.310305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.311219 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.311223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:59.312055 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.312065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.312488 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.313100 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:59.313116 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:59.313670 (+ 554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:59.316192 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.316195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.316197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.316385 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.316649 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.316652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.316653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.316700 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.317093 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:59.317096 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:59.317183 (+ 87us) client_negotiation.cc:769] Sending connection context
0128 15:02:59.317753 (+ 570us) client_negotiation.cc:240] Negotiation successful
0128 15:02:59.317896 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"spinlock_wait_cycles":4992,"thread_start_us":121,"threads_started":1}
I20260128 15:02:59.318871 21182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.283594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.188:44165 (local address 127.0.0.1:34044)
0128 15:02:59.283886 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:59.283899 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:59.284005 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:59.284252 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:59.284256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:59.308829 (+ 24573us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:59.309987 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.309998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.310920 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.310923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:59.312343 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.312353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.312755 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.313147 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:59.313164 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:59.313865 (+ 701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:59.316591 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.316594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.316597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.316792 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.317003 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.317006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.317009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.317048 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.317404 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:59.317407 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:59.317471 (+ 64us) client_negotiation.cc:769] Sending connection context
0128 15:02:59.318597 (+ 1126us) client_negotiation.cc:240] Negotiation successful
0128 15:02:59.318717 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"spinlock_wait_cycles":768,"thread_start_us":105,"threads_started":1}
I20260128 15:02:59.319159 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.283420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.13.88.190:41873)
0128 15:02:59.283529 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:59.283533 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:59.283863 (+ 330us) server_negotiation.cc:408] Connection header received
0128 15:02:59.283971 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:59.283974 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:59.284021 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:59.284092 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:59.309805 (+ 25713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.310486 (+ 681us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.311268 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.311444 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.313958 (+ 2514us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:59.313973 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:59.313976 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:59.314002 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:59.315662 (+ 1660us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.316129 (+ 467us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.316133 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.316135 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.316187 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.316463 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.316467 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.316468 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.316645 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:59.316708 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:59.318908 (+ 2200us) server_negotiation.cc:300] Negotiation successful
0128 15:02:59.319016 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260128 15:02:59.319522 21181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.283435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:56128)
0128 15:02:59.283780 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:59.283793 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:59.283890 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:59.284201 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:59.284204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:59.309408 (+ 25204us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:59.309684 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.309691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.310604 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.310608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:59.311154 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.311162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.311553 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.313353 (+ 1800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:59.313368 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:59.313844 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:59.315801 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.315805 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.315807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.316011 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.316283 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.316286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.316288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.316336 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.318553 (+ 2217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:59.318556 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:59.318633 (+ 77us) client_negotiation.cc:769] Sending connection context
0128 15:02:59.319270 (+ 637us) client_negotiation.cc:240] Negotiation successful
0128 15:02:59.319383 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":104,"threads_started":1}
Jan 28 15:02:59 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612577, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.190@KRBTEST.COM for oryx/127.13.88.188@KRBTEST.COM
I20260128 15:02:59.329643 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.322435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.188:44165 (local address 127.0.0.1:34054)
0128 15:02:59.322615 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:59.322626 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:59.322697 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:59.323103 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:59.323105 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:59.323279 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:59.323474 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.323482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.324224 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.324228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:59.324792 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.324802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.324902 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.325407 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:59.325423 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:59.326701 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:59.328555 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.328558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.328560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.328721 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.328933 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.328935 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.328937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.328978 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.329282 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:59.329284 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:59.329357 (+ 73us) client_negotiation.cc:769] Sending connection context
0128 15:02:59.329431 (+ 74us) client_negotiation.cc:240] Negotiation successful
0128 15:02:59.329527 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260128 15:02:59.329739 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.322532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34054 (local address 127.13.88.188:44165)
0128 15:02:59.322668 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:59.322673 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:59.322687 (+ 14us) server_negotiation.cc:408] Connection header received
0128 15:02:59.322826 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:59.322830 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:59.322882 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:59.322962 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:59.323566 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.324130 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.325090 (+ 960us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.325276 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.326811 (+ 1535us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:59.326828 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:59.326832 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:59.326857 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:59.328410 (+ 1553us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.328807 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.328811 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.328813 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.328858 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.329046 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.329048 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.329050 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.329200 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:59.329291 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:59.329488 (+ 197us) server_negotiation.cc:300] Negotiation successful
0128 15:02:59.329592 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260128 15:02:59.330669 20428 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.13.88.188:44165: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c103d7f4e85045059b10b02c5788eaba"
member_type: NON_VOTER
last_known_addr {
host: "127.13.88.188"
port: 44165
}
attrs {
promote: true
}
}
dest_uuid: "514b33d5af9e48e49897a8adce27696b"
cas_config_opid_index: 5
I20260128 15:02:59.331213 20428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: NON_VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: true } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } }
I20260128 15:02:59.332216 20999 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c103d7f4e85045059b10b02c5788eaba (127.13.88.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: NON_VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: true } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } } }
I20260128 15:02:59.332698 21050 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 LEARNER]: Refusing update from remote peer 514b33d5af9e48e49897a8adce27696b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260128 15:02:59.333101 21185 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: NON_VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260128 15:02:59.333642 20999 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.13.88.188:44165
I20260128 15:02:59.333817 20999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "514b33d5af9e48e49897a8adce27696b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: NON_VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: true } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } } }
I20260128 15:02:59.333899 20999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:59.334251 21012 master.cc:561] Master@127.13.88.188:44165 shutting down...
I20260128 15:02:59.335170 21050 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c103d7f4e85045059b10b02c5788eaba (127.13.88.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: NON_VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: true } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } } }
W20260128 15:02:59.336294 20413 consensus_peers.cc:597] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b -> Peer c103d7f4e85045059b10b02c5788eaba (127.13.88.188:44165): Couldn't send request to peer c103d7f4e85045059b10b02c5788eaba. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260128 15:02:59.337359 21079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 514b33d5af9e48e49897a8adce27696b. Latest consensus state: current_term: 1 leader_uuid: "514b33d5af9e48e49897a8adce27696b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: NON_VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: true } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } } }
I20260128 15:02:59.337450 21079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: This master's current role is: LEARNER
I20260128 15:02:59.337524 21079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "514b33d5af9e48e49897a8adce27696b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: NON_VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: true } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } } }
I20260128 15:02:59.337577 21079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: This master's current role is: LEARNER
I20260128 15:02:59.338104 21185 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b: attempting to promote NON_VOTER 1c36b3bafb1547efb54a35126ed2c62c to VOTER
I20260128 15:02:59.338541 21185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } }
W20260128 15:02:59.339038 20428 master.cc:679] Remote error: unable to get registration information for peer c103d7f4e85045059b10b02c5788eaba: Service unavailable: service kudu.master.MasterService not registered on Master
W20260128 15:02:59.340523 20413 consensus_peers.cc:597] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b -> Peer c103d7f4e85045059b10b02c5788eaba (127.13.88.188:44165): Couldn't send request to peer c103d7f4e85045059b10b02c5788eaba. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260128 15:02:59.340876 21050 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 LEARNER]: Refusing update from remote peer 514b33d5af9e48e49897a8adce27696b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260128 15:02:59.341287 21012 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 0 FOLLOWER]: Raft consensus shutting down.
I20260128 15:02:59.341271 21185 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260128 15:02:59.341568 21012 raft_consensus.cc:2272] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 0 FOLLOWER]: Raft consensus is shut down!
I20260128 15:02:59.341682 21012 tablet_replica.cc:333] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: stopping tablet replica
I20260128 15:02:59.343096 20998 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1c36b3bafb1547efb54a35126ed2c62c (127.13.88.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } } }
I20260128 15:02:59.343451 21050 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1c36b3bafb1547efb54a35126ed2c62c (127.13.88.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } } }
I20260128 15:02:59.344650 20998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "514b33d5af9e48e49897a8adce27696b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } } }
I20260128 15:02:59.344738 20998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:59.344872 20998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "514b33d5af9e48e49897a8adce27696b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } } }
I20260128 15:02:59.344944 20998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: This master's current role is: LEADER
I20260128 15:02:59.346174 21012 master.cc:583] Master@127.13.88.188:44165 shutdown complete.
I20260128 15:02:59.346215 21012 master_runner.cc:305] Clearing existing system tablet
I20260128 15:02:59.346396 21012 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:59.346439 21012 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:59.347359 21012 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260128 15:02:59.348058 21190 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:59.348278 21012 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:59.348382 21012 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:59.348441 21012 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/wal
uuid: "c103d7f4e85045059b10b02c5788eaba"
format_stamp: "Formatted at 2026-01-28 15:02:59 on dist-test-slave-m28t"
server_key: "9ecb1ef899e3c58a280cd645964c1cbb"
server_key_iv: "e266457e0c1f97690234d50cf78c3ade"
server_key_version: "encryptionkey@0"
I20260128 15:02:59.349906 21012 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:02:59.351449 21012 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260128 15:02:59.351521 21012 log.cc:1199] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260128 15:02:59.351706 21012 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: Deleting consensus metadata
I20260128 15:02:59.351830 21012 master_runner.cc:315] Copying system tablet from 127.13.88.190:41873
I20260128 15:02:59.352527 21187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "514b33d5af9e48e49897a8adce27696b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } } }
I20260128 15:02:59.352627 21187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: This master's current role is: FOLLOWER
I20260128 15:02:59.352880 21012 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: tablet copy: Beginning tablet copy session from remote peer at address 127.13.88.190:41873
I20260128 15:02:59.360477 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.353431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.13.88.190:41873)
0128 15:02:59.353566 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:59.353570 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:59.353648 (+ 78us) server_negotiation.cc:408] Connection header received
0128 15:02:59.353757 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:59.353761 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:59.353814 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:59.353909 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:02:59.354671 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.355486 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.356200 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.356404 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.357148 (+ 744us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:02:59.357168 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:02:59.357172 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:02:59.357195 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:02:59.358843 (+ 1648us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.359258 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.359262 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.359264 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.359315 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:02:59.359568 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:02:59.359572 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:02:59.359574 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:02:59.359752 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:02:59.359846 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:59.360207 (+ 361us) server_negotiation.cc:300] Negotiation successful
0128 15:02:59.360304 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260128 15:02:59.360507 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.353344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:56144)
0128 15:02:59.353571 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:59.353586 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:59.353681 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:59.353916 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:59.353918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:59.354083 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:02:59.354546 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.354554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.355628 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.355631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:59.356088 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.356097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.356209 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.356727 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:02:59.356740 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:02:59.357033 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:02:59.358964 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.358968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.358970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.359179 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.359405 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:02:59.359408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:02:59.359411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:02:59.359470 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:02:59.359934 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:02:59.359937 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:02:59.360034 (+ 97us) client_negotiation.cc:769] Sending connection context
0128 15:02:59.360147 (+ 113us) client_negotiation.cc:240] Negotiation successful
0128 15:02:59.360288 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
I20260128 15:02:59.360561 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.360191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.188:44165 (local address 127.0.0.1:34060)
0128 15:02:59.360361 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:59.360458 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.188:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260128 15:02:59.360954 20428 master.cc:679] Network error: unable to get registration information for peer c103d7f4e85045059b10b02c5788eaba: Client connection negotiation failed: client connection to 127.13.88.188:44165: connect: Connection refused (error 111)
I20260128 15:02:59.361006 20449 tablet_copy_service.cc:140] P 514b33d5af9e48e49897a8adce27696b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c103d7f4e85045059b10b02c5788eaba ({username='oryx', principal='oryx/127.13.88.188@KRBTEST.COM'} at 127.0.0.1:56144)
I20260128 15:02:59.361081 20449 tablet_copy_service.cc:161] P 514b33d5af9e48e49897a8adce27696b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c103d7f4e85045059b10b02c5788eaba at {username='oryx', principal='oryx/127.13.88.188@KRBTEST.COM'} at 127.0.0.1:56144: session id = c103d7f4e85045059b10b02c5788eaba-00000000000000000000000000000000
I20260128 15:02:59.361857 20449 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b: Tablet Copy: opened 0 blocks and 1 log segments
I20260128 15:02:59.362983 21012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260128 15:02:59.364637 21012 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: tablet copy: Starting download of 0 data blocks...
I20260128 15:02:59.364784 21012 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: tablet copy: Starting download of 1 WAL segments...
I20260128 15:02:59.366153 21012 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260128 15:02:59.367481 20449 tablet_copy_service.cc:342] P 514b33d5af9e48e49897a8adce27696b: Request end of tablet copy session c103d7f4e85045059b10b02c5788eaba-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.88.188@KRBTEST.COM'} at 127.0.0.1:56144
I20260128 15:02:59.367544 20449 tablet_copy_service.cc:434] P 514b33d5af9e48e49897a8adce27696b: ending tablet copy session c103d7f4e85045059b10b02c5788eaba-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c103d7f4e85045059b10b02c5788eaba
W20260128 15:02:59.371412 21012 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260128 15:02:59.373477 21012 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:02:59.374948 21197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:02:59.374996 21198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:59.375190 21012 server_base.cc:1060] running on GCE node
W20260128 15:02:59.380576 21200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:02:59.380831 21012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:02:59.381158 21012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:02:59.382370 21012 hybrid_clock.cc:648] HybridClock initialized: now 1769612579382356 us; error 33 us; skew 500 ppm
Jan 28 15:02:59 dist-test-slave-m28t krb5kdc[20381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612579, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:02:59.383769 21012 init.cc:377] Logged in from keytab as oryx/127.13.88.188@KRBTEST.COM (short username oryx)
I20260128 15:02:59.384277 21012 webserver.cc:492] Webserver started at http://127.13.88.188:33353/ using document root <none> and password file <none>
I20260128 15:02:59.384495 21012 fs_manager.cc:362] Metadata directory not provided
I20260128 15:02:59.384531 21012 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:02:59.385530 21012 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:59.386371 21207 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:02:59.386602 21012 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260128 15:02:59.386668 21012 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/wal
uuid: "c103d7f4e85045059b10b02c5788eaba"
format_stamp: "Formatted at 2026-01-28 15:02:59 on dist-test-slave-m28t"
server_key: "9ecb1ef899e3c58a280cd645964c1cbb"
server_key_iv: "e266457e0c1f97690234d50cf78c3ade"
server_key_version: "encryptionkey@0"
I20260128 15:02:59.386745 21012 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:02:59.396903 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.396618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.188:44165 (local address 127.0.0.1:34074)
0128 15:02:59.396748 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:59.396807 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.188:44165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260128 15:02:59.397112 20428 master.cc:679] Network error: unable to get registration information for peer c103d7f4e85045059b10b02c5788eaba: Client connection negotiation failed: client connection to 127.13.88.188:44165: connect: Connection refused (error 111)
I20260128 15:02:59.412024 21012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:02:59.413895 21012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:02:59.418584 21012 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.188:44165
I20260128 15:02:59.418604 21259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.188:44165 every 8 connection(s)
I20260128 15:02:59.419548 21012 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/master-2/data/info.pb
I20260128 15:02:59.421005 21260 sys_catalog.cc:263] Verifying existing consensus state
I20260128 15:02:59.421818 21260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: Bootstrap starting.
I20260128 15:02:59.431579 21260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260128 15:02:59.431921 21260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: Bootstrap complete.
I20260128 15:02:59.432699 21260 raft_consensus.cc:359] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } }
I20260128 15:02:59.432849 21260 raft_consensus.cc:740] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c103d7f4e85045059b10b02c5788eaba, State: Initialized, Role: LEARNER
I20260128 15:02:59.432965 21260 consensus_queue.cc:260] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } }
I20260128 15:02:59.433419 21263 sys_catalog.cc:455] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } } }
I20260128 15:02:59.433655 21263 sys_catalog.cc:458] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [sys.catalog]: This master's current role is: LEARNER
I20260128 15:02:59.433501 21260 sys_catalog.cc:565] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:02:59.438656 21274 catalog_manager.cc:1269] Loaded cluster ID: 160ee1f718a440ada4dbc6a39dc86fae
I20260128 15:02:59.438719 21274 catalog_manager.cc:1562] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: loading cluster ID for follower catalog manager: success
I20260128 15:02:59.440372 21274 catalog_manager.cc:1584] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: acquiring CA information for follower catalog manager: success
I20260128 15:02:59.441648 21274 catalog_manager.cc:1612] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260128 15:02:59.467150 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.464628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.188:44165 (local address 127.0.0.1:34080)
0128 15:02:59.464763 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0128 15:02:59.464776 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:02:59.464846 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:02:59.465164 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:02:59.465166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:02:59.465180 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:02:59.465367 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.465372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.466239 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.466242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:02:59.466835 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:02:59.466843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.466943 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.466962 (+ 19us) client_negotiation.cc:769] Sending connection context
0128 15:02:59.466996 (+ 34us) client_negotiation.cc:240] Negotiation successful
0128 15:02:59.467041 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260128 15:02:59.467617 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:02:59.464726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.13.88.188:44165)
0128 15:02:59.464952 (+ 226us) server_negotiation.cc:207] Beginning negotiation
0128 15:02:59.464956 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:02:59.464968 (+ 12us) server_negotiation.cc:408] Connection header received
0128 15:02:59.465011 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:02:59.465014 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:02:59.465068 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:02:59.465151 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:02:59.465454 (+ 303us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.466124 (+ 670us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:02:59.466937 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:02:59.467273 (+ 336us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:02:59.467424 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
0128 15:02:59.467467 (+ 43us) server_negotiation.cc:300] Negotiation successful
0128 15:02:59.467518 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":76,"threads_started":1}
I20260128 15:02:59.499939 20758 heartbeater.cc:499] Master 127.13.88.190:41873 was elected leader, sending a full tablet report...
I20260128 15:02:59.719604 20894 heartbeater.cc:499] Master 127.13.88.190:41873 was elected leader, sending a full tablet report...
I20260128 15:02:59.749980 21234 raft_consensus.cc:1217] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260128 15:02:59.750434 21263 sys_catalog.cc:455] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 514b33d5af9e48e49897a8adce27696b. Latest consensus state: current_term: 1 leader_uuid: "514b33d5af9e48e49897a8adce27696b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } } }
I20260128 15:02:59.750557 21263 sys_catalog.cc:458] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [sys.catalog]: This master's current role is: LEARNER
I20260128 15:03:00.365535 21276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "514b33d5af9e48e49897a8adce27696b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: NON_VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: true } } }
I20260128 15:03:00.365689 21276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:00.366564 21276 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b: attempting to promote NON_VOTER c103d7f4e85045059b10b02c5788eaba to VOTER
I20260128 15:03:00.366904 21276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } }
I20260128 15:03:00.368112 21050 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 FOLLOWER]: Refusing update from remote peer 514b33d5af9e48e49897a8adce27696b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260128 15:03:00.368338 21234 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 1 LEARNER]: Refusing update from remote peer 514b33d5af9e48e49897a8adce27696b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260128 15:03:00.368547 21276 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260128 15:03:00.368734 21278 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260128 15:03:00.370622 21277 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c103d7f4e85045059b10b02c5788eaba (127.13.88.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } } }
I20260128 15:03:00.371148 21050 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c103d7f4e85045059b10b02c5788eaba (127.13.88.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } } }
I20260128 15:03:00.371940 21277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "514b33d5af9e48e49897a8adce27696b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } } }
I20260128 15:03:00.372025 21277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:00.372148 21277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "514b33d5af9e48e49897a8adce27696b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } } }
I20260128 15:03:00.372196 21277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:00.372732 21283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "514b33d5af9e48e49897a8adce27696b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } } }
I20260128 15:03:00.372843 21283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: This master's current role is: FOLLOWER
I20260128 15:03:00.373340 21234 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c103d7f4e85045059b10b02c5788eaba (127.13.88.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } } }
I20260128 15:03:00.375137 21277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "514b33d5af9e48e49897a8adce27696b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } } }
I20260128 15:03:00.375217 21277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:00.375483 21284 sys_catalog.cc:455] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "514b33d5af9e48e49897a8adce27696b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } } }
I20260128 15:03:00.375594 21284 sys_catalog.cc:458] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [sys.catalog]: This master's current role is: FOLLOWER
I20260128 15:03:00.383535 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.376671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57738 (local address 127.13.88.189:38211)
0128 15:03:00.376925 (+ 254us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.376930 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.376946 (+ 16us) server_negotiation.cc:408] Connection header received
0128 15:03:00.377037 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.377041 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.377089 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.377180 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:00.377876 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.378722 (+ 846us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.379311 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.379512 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.380410 (+ 898us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:00.380456 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:00.380459 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:00.380483 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:00.382189 (+ 1706us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.382552 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.382555 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.382557 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.382603 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.382762 (+ 159us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.382765 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.382767 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.382937 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:00.383061 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.383211 (+ 150us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.383341 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260128 15:03:00.386801 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.384328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:56146)
0128 15:03:00.384556 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:00.384568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:00.384647 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:00.384895 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:00.384899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:00.384915 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:00.385084 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.385090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.385881 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.385885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:00.386446 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.386453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.386549 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.386567 (+ 18us) client_negotiation.cc:769] Sending connection context
0128 15:03:00.386612 (+ 45us) client_negotiation.cc:240] Negotiation successful
0128 15:03:00.386668 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":96,"threads_started":1}
I20260128 15:03:00.387228 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.384452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56146 (local address 127.13.88.190:41873)
0128 15:03:00.384723 (+ 271us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.384727 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.384739 (+ 12us) server_negotiation.cc:408] Connection header received
0128 15:03:00.384774 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.384776 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.384814 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.384878 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:00.385184 (+ 306us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.385784 (+ 600us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.386550 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.386995 (+ 445us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.387032 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.387072 (+ 40us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.387137 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":105,"threads_started":1}
I20260128 15:03:00.391438 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.389190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.189:38211 (local address 127.0.0.1:57750)
0128 15:03:00.389283 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:00.389295 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:00.389353 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:00.389552 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:00.389554 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:00.389576 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:00.389712 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.389716 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.390646 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.390649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:00.391186 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.391192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.391272 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.391285 (+ 13us) client_negotiation.cc:769] Sending connection context
0128 15:03:00.391316 (+ 31us) client_negotiation.cc:240] Negotiation successful
0128 15:03:00.391357 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"spinlock_wait_cycles":896}
I20260128 15:03:00.392032 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.389252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.13.88.189:38211)
0128 15:03:00.389370 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.389374 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.389388 (+ 14us) server_negotiation.cc:408] Connection header received
0128 15:03:00.389426 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.389429 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.389472 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.389548 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:00.389792 (+ 244us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.390550 (+ 758us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.391285 (+ 735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.391797 (+ 512us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.391830 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.391874 (+ 44us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.391928 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260128 15:03:00.395272 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.393031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.188:44165 (local address 127.0.0.1:34090)
0128 15:03:00.393165 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:00.393175 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:00.393228 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:00.393588 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:00.393591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:00.393606 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:00.393739 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.393743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.394412 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.394416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:00.394938 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.394944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.395039 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.395060 (+ 21us) client_negotiation.cc:769] Sending connection context
0128 15:03:00.395114 (+ 54us) client_negotiation.cc:240] Negotiation successful
0128 15:03:00.395166 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260128 15:03:00.395746 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.393153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34090 (local address 127.13.88.188:44165)
0128 15:03:00.393410 (+ 257us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.393413 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.393424 (+ 11us) server_negotiation.cc:408] Connection header received
0128 15:03:00.393461 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.393463 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.393500 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.393568 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:00.393832 (+ 264us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.394313 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.395043 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.395536 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.395566 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.395607 (+ 41us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.395652 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
I20260128 15:03:00.403468 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.397458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.13.88.188:44165)
0128 15:03:00.397744 (+ 286us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.397747 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.397756 (+ 9us) server_negotiation.cc:408] Connection header received
0128 15:03:00.397780 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.397782 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.397825 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.397887 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:00.398596 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.399029 (+ 433us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.399586 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.399751 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.400802 (+ 1051us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:00.400817 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:00.400819 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:00.400841 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:00.402197 (+ 1356us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.402535 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.402538 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.402539 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.402581 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.402763 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.402766 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.402768 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.402906 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:00.403021 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.403214 (+ 193us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.403322 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260128 15:03:00.406700 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.404144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:56152)
0128 15:03:00.404369 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:00.404380 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:00.404478 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:00.404710 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:00.404712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:00.404722 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:00.404887 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.404891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.405875 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.405878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:00.406438 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.406445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.406529 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.406541 (+ 12us) client_negotiation.cc:769] Sending connection context
0128 15:03:00.406572 (+ 31us) client_negotiation.cc:240] Negotiation successful
0128 15:03:00.406612 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260128 15:03:00.407353 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.404254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.13.88.190:41873)
0128 15:03:00.404404 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.404408 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.404444 (+ 36us) server_negotiation.cc:408] Connection header received
0128 15:03:00.404569 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.404572 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.404622 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.404708 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:00.404969 (+ 261us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.405779 (+ 810us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.406550 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.407067 (+ 517us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.407110 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.407158 (+ 48us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.407214 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260128 15:03:00.410441 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.408337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.189:38211 (local address 127.0.0.1:57756)
0128 15:03:00.408454 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:00.408465 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:00.408521 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:00.408773 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:00.408776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:00.408793 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:00.408927 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.408932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.409585 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.409589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:00.410118 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.410124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.410217 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.410233 (+ 16us) client_negotiation.cc:769] Sending connection context
0128 15:03:00.410278 (+ 45us) client_negotiation.cc:240] Negotiation successful
0128 15:03:00.410338 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260128 15:03:00.410848 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.408483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.13.88.189:38211)
0128 15:03:00.408610 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.408615 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.408627 (+ 12us) server_negotiation.cc:408] Connection header received
0128 15:03:00.408661 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.408664 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.408700 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.408758 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:00.409021 (+ 263us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.409493 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.410218 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.410650 (+ 432us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.410681 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.410719 (+ 38us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.410760 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260128 15:03:00.414021 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.411877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.188:44165 (local address 127.0.0.1:34110)
0128 15:03:00.411959 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:00.411972 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:00.412045 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:00.412333 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:00.412335 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:00.412362 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:00.412542 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.412548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.413164 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.413168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:00.413700 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.413706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.413802 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.413816 (+ 14us) client_negotiation.cc:769] Sending connection context
0128 15:03:00.413863 (+ 47us) client_negotiation.cc:240] Negotiation successful
0128 15:03:00.413911 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"spinlock_wait_cycles":3328}
I20260128 15:03:00.414407 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.411953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34110 (local address 127.13.88.188:44165)
0128 15:03:00.412053 (+ 100us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.412056 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.412069 (+ 13us) server_negotiation.cc:408] Connection header received
0128 15:03:00.412204 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.412208 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.412246 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.412320 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:00.412640 (+ 320us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.413077 (+ 437us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.413799 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.414210 (+ 411us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.414240 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.414292 (+ 52us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.414337 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44,"spinlock_wait_cycles":1408}
I20260128 15:03:00.415594 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 20493
I20260128 15:03:00.422577 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:33663
--local_ip_for_outbound_sockets=127.13.88.129
--tserver_master_addrs=127.13.88.190:41873
--webserver_port=43729
--webserver_interface=127.13.88.129
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.88.190:41873,127.13.88.189:38211
--tserver_master_addrs=127.13.88.190:41873,127.13.88.189:38211,127.13.88.188:44165 with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:03:00.542207 21292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:00.542431 21292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:00.542498 21292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:00.545766 21292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:00.545833 21292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:00.545933 21292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:03:00.550244 21292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:33663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=43729
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:41873,127.13.88.189:38211,127.13.88.188:44165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:00.551491 21292 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:00.552393 21292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:00.558986 21300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:00.558985 21297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:00.559038 21298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:00.559257 21292 server_base.cc:1060] running on GCE node
I20260128 15:03:00.559711 21292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:00.560258 21292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:00.561412 21292 hybrid_clock.cc:648] HybridClock initialized: now 1769612580561390 us; error 35 us; skew 500 ppm
Jan 28 15:03:00 dist-test-slave-m28t krb5kdc[20381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612580, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:00.563959 21292 init.cc:377] Logged in from keytab as oryx/127.13.88.129@KRBTEST.COM (short username oryx)
I20260128 15:03:00.564970 21292 webserver.cc:492] Webserver started at http://127.13.88.129:43729/ using document root <none> and password file <none>
I20260128 15:03:00.565493 21292 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:00.565546 21292 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:00.568895 21292 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260128 15:03:00.570639 21307 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:00.571686 21292 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:03:00.571779 21292 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "3fc2239548eb481580722ea108a90e21"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "fdf8de574eb2e3d1686576954215ef2f"
server_key_iv: "7ad0f8691be46644d2cc6dbef43ca769"
server_key_version: "encryptionkey@0"
I20260128 15:03:00.572178 21292 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:00.588488 21292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:00.590953 21292 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:00.591122 21292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:00.591622 21292 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:00.592537 21292 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:00.592584 21292 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:00.592623 21292 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:00.592643 21292 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:00.601243 21292 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:33663
I20260128 15:03:00.601269 21420 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:33663 every 8 connection(s)
I20260128 15:03:00.602267 21292 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:03:00.608821 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 21292
I20260128 15:03:00.608989 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 20629
Jan 28 15:03:00 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612580, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM
Jan 28 15:03:00 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612580, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.188@KRBTEST.COM
Jan 28 15:03:00 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612580, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.189@KRBTEST.COM
I20260128 15:03:00.616367 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.605283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:42025 (local address 127.13.88.189:38211)
0128 15:03:00.605398 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.605401 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.605486 (+ 85us) server_negotiation.cc:408] Connection header received
0128 15:03:00.605919 (+ 433us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.605923 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.605975 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.606025 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:00.607400 (+ 1375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.607885 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.608648 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.608826 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.612367 (+ 3541us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:00.612388 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:00.612391 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:00.612440 (+ 49us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:00.614053 (+ 1613us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.614745 (+ 692us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.614749 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.614751 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.614800 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.615142 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.615145 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.615147 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.615320 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:00.615387 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.616088 (+ 701us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.616209 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260128 15:03:00.616374 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.603930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:56061 (local address 127.13.88.188:44165)
0128 15:03:00.604072 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.604076 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.604804 (+ 728us) server_negotiation.cc:408] Connection header received
0128 15:03:00.605919 (+ 1115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.605923 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.605975 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.606025 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:00.607446 (+ 1421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.608150 (+ 704us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.608831 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.609018 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.612390 (+ 3372us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:00.612410 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:00.612413 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:00.612473 (+ 60us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:00.614190 (+ 1717us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.614773 (+ 583us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.614776 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.614779 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.614827 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.615129 (+ 302us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.615133 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.615135 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.615311 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:00.615389 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.616083 (+ 694us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.616209 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260128 15:03:00.617192 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.603977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:46677 (local address 127.13.88.190:41873)
0128 15:03:00.604123 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.604128 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.604807 (+ 679us) server_negotiation.cc:408] Connection header received
0128 15:03:00.605897 (+ 1090us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.605901 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.605961 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.606032 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:00.607335 (+ 1303us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.608081 (+ 746us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.609213 (+ 1132us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.609379 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.612110 (+ 2731us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:00.612131 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:00.612134 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:00.612164 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:00.614472 (+ 2308us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.614934 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.614938 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.614940 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.615004 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.615238 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.615241 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.615243 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.615411 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:00.615487 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.616920 (+ 1433us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.617057 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260128 15:03:00.617486 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.604237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.188:44165 (local address 127.13.88.129:56061)
0128 15:03:00.604670 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:00.604714 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:00.605719 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:00.606196 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:00.606204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:00.607097 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:00.607340 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.607347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.608261 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.608264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:00.608701 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.608710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.609132 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.609903 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:00.609920 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:00.611900 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:00.614312 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.614315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.614318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.614668 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.614920 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.614923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.614925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.615032 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.615465 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:00.615471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:00.615777 (+ 306us) client_negotiation.cc:769] Sending connection context
0128 15:03:00.616582 (+ 805us) client_negotiation.cc:240] Negotiation successful
0128 15:03:00.616720 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":38,"spinlock_wait_cycles":46336,"thread_start_us":141,"threads_started":1}
I20260128 15:03:00.617499 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.605190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.189:38211 (local address 127.13.88.129:42025)
0128 15:03:00.605411 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:00.605422 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:00.605729 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:00.606196 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:00.606204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:00.606657 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:00.607251 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.607259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.607997 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.608000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:00.608522 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.608531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.608939 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.609771 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:00.609807 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:00.612248 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:00.614201 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.614207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.614219 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.614643 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.614902 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.614905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.614944 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.615038 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.615465 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:00.615471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:00.615778 (+ 307us) client_negotiation.cc:769] Sending connection context
0128 15:03:00.616598 (+ 820us) client_negotiation.cc:240] Negotiation successful
0128 15:03:00.616791 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":58,"spinlock_wait_cycles":20480,"thread_start_us":62,"threads_started":1}
I20260128 15:03:00.618018 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.604237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.129:46677)
0128 15:03:00.604670 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:00.604714 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:00.605721 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:00.606193 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:00.606201 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:00.606634 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:00.607197 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.607209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.608194 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.608198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:00.608755 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.608763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.608933 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.609770 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:00.609813 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:00.611995 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:00.614588 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.614591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.614594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.614840 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.615102 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.615105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.615107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.615152 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.615522 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:00.615525 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:00.615778 (+ 253us) client_negotiation.cc:769] Sending connection context
0128 15:03:00.616005 (+ 227us) client_negotiation.cc:240] Negotiation successful
0128 15:03:00.616289 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"spinlock_wait_cycles":86400,"thread_start_us":119,"threads_started":1}
I20260128 15:03:00.618373 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:41581
--local_ip_for_outbound_sockets=127.13.88.130
--tserver_master_addrs=127.13.88.190:41873
--webserver_port=34791
--webserver_interface=127.13.88.130
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.88.190:41873,127.13.88.189:38211
--tserver_master_addrs=127.13.88.190:41873,127.13.88.189:38211,127.13.88.188:44165 with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260128 15:03:00.618719 21421 heartbeater.cc:344] Connected to a master server at 127.13.88.188:44165
I20260128 15:03:00.618943 21421 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:00.619241 21422 heartbeater.cc:344] Connected to a master server at 127.13.88.190:41873
I20260128 15:03:00.619324 21422 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:00.619583 21422 heartbeater.cc:507] Master 127.13.88.190:41873 requested a full tablet report, sending...
I20260128 15:03:00.619583 21421 heartbeater.cc:507] Master 127.13.88.188:44165 requested a full tablet report, sending...
I20260128 15:03:00.619758 21423 heartbeater.cc:344] Connected to a master server at 127.13.88.189:38211
I20260128 15:03:00.619853 21423 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:00.620047 21423 heartbeater.cc:507] Master 127.13.88.189:38211 requested a full tablet report, sending...
I20260128 15:03:00.620862 20428 ts_manager.cc:194] Re-registered known tserver with Master: 3fc2239548eb481580722ea108a90e21 (127.13.88.129:33663)
I20260128 15:03:00.621122 21224 ts_manager.cc:194] Registered new tserver with Master: 3fc2239548eb481580722ea108a90e21 (127.13.88.129:33663)
I20260128 15:03:00.621120 21040 ts_manager.cc:194] Registered new tserver with Master: 3fc2239548eb481580722ea108a90e21 (127.13.88.129:33663)
I20260128 15:03:00.621469 20428 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.88.129@KRBTEST.COM'} at 127.13.88.129:46677
W20260128 15:03:00.723595 21428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:00.723822 21428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:00.723878 21428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:00.727108 21428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:00.727164 21428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:00.727244 21428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:03:00.731325 21428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:41581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=34791
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:41873,127.13.88.189:38211,127.13.88.188:44165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:00.732314 21428 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:00.733111 21428 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:00.739358 21433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:00.739368 21434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:00.739338 21436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:00.739643 21428 server_base.cc:1060] running on GCE node
I20260128 15:03:00.739979 21428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:00.740520 21428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:00.741665 21428 hybrid_clock.cc:648] HybridClock initialized: now 1769612580741647 us; error 38 us; skew 500 ppm
Jan 28 15:03:00 dist-test-slave-m28t krb5kdc[20381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612580, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:00.744194 21428 init.cc:377] Logged in from keytab as oryx/127.13.88.130@KRBTEST.COM (short username oryx)
I20260128 15:03:00.745204 21428 webserver.cc:492] Webserver started at http://127.13.88.130:34791/ using document root <none> and password file <none>
I20260128 15:03:00.745709 21428 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:00.745764 21428 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:00.749050 21428 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260128 15:03:00.750815 21443 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:00.751843 21428 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:03:00.751940 21428 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "23361dc3c81bafe02befe28c1219e49b"
server_key_iv: "35d1c3eb0e15dd426b6a483efdd51952"
server_key_version: "encryptionkey@0"
I20260128 15:03:00.752337 21428 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:00.767079 21428 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:00.769663 21428 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:00.769830 21428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:00.770361 21428 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:00.771230 21428 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:00.771278 21428 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:00.771329 21428 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:00.771358 21428 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:00.780727 21428 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:41581
I20260128 15:03:00.780769 21556 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:41581 every 8 connection(s)
I20260128 15:03:00.781700 21428 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
I20260128 15:03:00.785204 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 21428
I20260128 15:03:00.785369 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 20765
Jan 28 15:03:00 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612580, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.189@KRBTEST.COM
Jan 28 15:03:00 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612580, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.188@KRBTEST.COM
Jan 28 15:03:00 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612580, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM
I20260128 15:03:00.793115 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:45859
--local_ip_for_outbound_sockets=127.13.88.131
--tserver_master_addrs=127.13.88.190:41873
--webserver_port=34479
--webserver_interface=127.13.88.131
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.88.190:41873,127.13.88.189:38211
--tserver_master_addrs=127.13.88.190:41873,127.13.88.189:38211,127.13.88.188:44165 with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260128 15:03:00.796834 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.783322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:54661 (local address 127.13.88.188:44165)
0128 15:03:00.783471 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.783476 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.784794 (+ 1318us) server_negotiation.cc:408] Connection header received
0128 15:03:00.786315 (+ 1521us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.786318 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.786377 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.786450 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:00.787723 (+ 1273us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.788444 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.789312 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.789497 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.793386 (+ 3889us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:00.793404 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:00.793406 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:00.793433 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:00.795024 (+ 1591us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.795567 (+ 543us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.795571 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.795573 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.795625 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.795845 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.795848 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.795850 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.795991 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:00.796093 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.796560 (+ 467us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.796677 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260128 15:03:00.796859 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.784766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:54479 (local address 127.13.88.190:41873)
0128 15:03:00.784884 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.784888 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.785274 (+ 386us) server_negotiation.cc:408] Connection header received
0128 15:03:00.788264 (+ 2990us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.788268 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.788327 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.788388 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:00.789103 (+ 715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.789862 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.790680 (+ 818us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.790861 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.793093 (+ 2232us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:00.793113 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:00.793116 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:00.793148 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:00.794769 (+ 1621us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.795402 (+ 633us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.795405 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.795407 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.795457 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.795707 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.795710 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.795712 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.795883 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:00.795951 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.796582 (+ 631us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.796702 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260128 15:03:00.797504 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.783898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:59373 (local address 127.13.88.189:38211)
0128 15:03:00.784007 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.784011 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.784980 (+ 969us) server_negotiation.cc:408] Connection header received
0128 15:03:00.786434 (+ 1454us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.786437 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.786492 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.786556 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:00.788138 (+ 1582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.788885 (+ 747us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.789934 (+ 1049us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.790114 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.793045 (+ 2931us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:00.793065 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:00.793069 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:00.793106 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:00.794710 (+ 1604us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.795338 (+ 628us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.795342 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.795344 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.795397 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.795707 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.795710 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.795712 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.795883 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:00.795962 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.797249 (+ 1287us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.797371 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260128 15:03:00.797674 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.783671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.189:38211 (local address 127.13.88.130:59373)
0128 15:03:00.784669 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:00.784917 (+ 248us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:00.786223 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:00.786997 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:00.787009 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:00.787751 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:00.788027 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.788035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.789006 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.789009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:00.789531 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.789539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.789725 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.790499 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:00.790542 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:00.792902 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:00.794834 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.794840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.794853 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.795242 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.795499 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.795502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.795504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.795620 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.796030 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:00.796038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:00.796315 (+ 277us) client_negotiation.cc:769] Sending connection context
0128 15:03:00.796521 (+ 206us) client_negotiation.cc:240] Negotiation successful
0128 15:03:00.796832 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":133,"spinlock_wait_cycles":43904,"thread_start_us":113,"threads_started":1}
I20260128 15:03:00.797897 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.783672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.188:44165 (local address 127.13.88.130:54661)
0128 15:03:00.784668 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:00.784714 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:00.786121 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:00.786620 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:00.786629 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:00.787008 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:00.787587 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.787599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.788580 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.788584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:00.789194 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.789201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.789725 (+ 524us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.790499 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:00.790542 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:00.793232 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:00.795109 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.795112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.795114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.795465 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.795725 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.795727 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.795729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.795768 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.796090 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:00.796093 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:00.796302 (+ 209us) client_negotiation.cc:769] Sending connection context
0128 15:03:00.796522 (+ 220us) client_negotiation.cc:240] Negotiation successful
0128 15:03:00.797750 (+ 1228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"spinlock_wait_cycles":10112,"thread_start_us":118,"threads_started":1}
I20260128 15:03:00.797669 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.784696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.130:54479)
0128 15:03:00.785181 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:00.785194 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:00.788161 (+ 2967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:00.788544 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:00.788547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:00.788706 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:00.788980 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.788986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.789990 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.789993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:00.790556 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.790564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.790969 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.791399 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:00.791414 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:00.792958 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:00.794893 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.794897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.794899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.795287 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.795552 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.795555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.795557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.795610 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.796030 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:00.796038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:00.796315 (+ 277us) client_negotiation.cc:769] Sending connection context
0128 15:03:00.797044 (+ 729us) client_negotiation.cc:240] Negotiation successful
0128 15:03:00.797161 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"spinlock_wait_cycles":12928,"thread_start_us":73,"threads_started":1}
I20260128 15:03:00.798870 21559 heartbeater.cc:344] Connected to a master server at 127.13.88.189:38211
I20260128 15:03:00.798918 21558 heartbeater.cc:344] Connected to a master server at 127.13.88.190:41873
I20260128 15:03:00.799145 21558 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:00.799145 21559 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:00.799757 21559 heartbeater.cc:507] Master 127.13.88.189:38211 requested a full tablet report, sending...
I20260128 15:03:00.799757 21558 heartbeater.cc:507] Master 127.13.88.190:41873 requested a full tablet report, sending...
I20260128 15:03:00.798870 21557 heartbeater.cc:344] Connected to a master server at 127.13.88.188:44165
I20260128 15:03:00.800410 21557 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:00.800948 21040 ts_manager.cc:194] Registered new tserver with Master: e5e25f3cc9ae44aba196ed92a3adfb0e (127.13.88.130:41581)
I20260128 15:03:00.800961 20428 ts_manager.cc:194] Re-registered known tserver with Master: e5e25f3cc9ae44aba196ed92a3adfb0e (127.13.88.130:41581)
I20260128 15:03:00.800630 21557 heartbeater.cc:507] Master 127.13.88.188:44165 requested a full tablet report, sending...
I20260128 15:03:00.801643 20428 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.88.130@KRBTEST.COM'} at 127.13.88.130:54479
I20260128 15:03:00.801934 21224 ts_manager.cc:194] Registered new tserver with Master: e5e25f3cc9ae44aba196ed92a3adfb0e (127.13.88.130:41581)
W20260128 15:03:00.902381 21564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:00.902592 21564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:00.902647 21564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:00.905910 21564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:00.905972 21564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:00.906060 21564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:03:00.910328 21564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:45859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=34479
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:41873,127.13.88.189:38211,127.13.88.188:44165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:00.911322 21564 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:00.912147 21564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260128 15:03:00.918613 21564 server_base.cc:1060] running on GCE node
W20260128 15:03:00.918545 21569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:00.918567 21572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:00.918545 21570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:00.919147 21564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:00.919636 21564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:00.920768 21564 hybrid_clock.cc:648] HybridClock initialized: now 1769612580920756 us; error 25 us; skew 500 ppm
Jan 28 15:03:00 dist-test-slave-m28t krb5kdc[20381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612580, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:00.923236 21564 init.cc:377] Logged in from keytab as oryx/127.13.88.131@KRBTEST.COM (short username oryx)
I20260128 15:03:00.924266 21564 webserver.cc:492] Webserver started at http://127.13.88.131:34479/ using document root <none> and password file <none>
I20260128 15:03:00.924824 21564 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:00.924886 21564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:00.928290 21564 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260128 15:03:00.930202 21579 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:00.931277 21564 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:03:00.931384 21564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "fc94066dcfc943bdb867b2d77a4b8efe"
format_stamp: "Formatted at 2026-01-28 15:02:58 on dist-test-slave-m28t"
server_key: "f294204dbe9f77c29e162123250154c0"
server_key_iv: "2aad845b945a96eb5accbbf8ab2ee310"
server_key_version: "encryptionkey@0"
I20260128 15:03:00.931806 21564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:00.951324 21564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:00.953826 21564 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:00.953994 21564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:00.954541 21564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:00.955433 21564 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:00.955484 21564 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:00.955528 21564 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:00.955559 21564 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:00.964162 21564 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:45859
I20260128 15:03:00.964192 21692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:45859 every 8 connection(s)
I20260128 15:03:00.965157 21564 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
I20260128 15:03:00.972446 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 21564
Jan 28 15:03:00 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612580, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM
Jan 28 15:03:00 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612580, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.189@KRBTEST.COM
Jan 28 15:03:00 dist-test-slave-m28t krb5kdc[20381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612580, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.188@KRBTEST.COM
I20260128 15:03:00.978571 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.966917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:44753 (local address 127.13.88.188:44165)
0128 15:03:00.967040 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.967045 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.967664 (+ 619us) server_negotiation.cc:408] Connection header received
0128 15:03:00.968798 (+ 1134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.968802 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.968853 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.968903 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:00.970326 (+ 1423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.971103 (+ 777us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.971938 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.972957 (+ 1019us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.975247 (+ 2290us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:00.975264 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:00.975268 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:00.975288 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:00.976995 (+ 1707us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.977440 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.977444 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.977446 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.977490 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.977704 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.977708 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.977710 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.977861 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:00.977946 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.978311 (+ 365us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.978416 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260128 15:03:00.979168 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.967087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.131:52677)
0128 15:03:00.967528 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:00.967570 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:00.968580 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:00.969068 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:00.969075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:00.969487 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:00.970166 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.970180 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.971134 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.971136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:00.971520 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.971526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.971718 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.972320 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:00.972354 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:00.974111 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:00.976327 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.976333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.976354 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.976684 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.976954 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.976958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.976959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.977082 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.977530 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:00.977536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:00.977823 (+ 287us) client_negotiation.cc:769] Sending connection context
0128 15:03:00.978041 (+ 218us) client_negotiation.cc:240] Negotiation successful
0128 15:03:00.978327 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":20,"spinlock_wait_cycles":57984,"thread_start_us":100,"threads_started":1}
I20260128 15:03:00.979185 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.967086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.188:44165 (local address 127.13.88.131:44753)
0128 15:03:00.967528 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:00.967570 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:00.968577 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:00.969070 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:00.969077 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:00.969942 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:00.970191 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.970197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.971225 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.971228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:00.971813 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.971821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.972505 (+ 684us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.973217 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:00.973234 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:00.975146 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:00.977148 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.977151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.977153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.977346 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.977571 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.977574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.977576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.977621 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.977987 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:00.977991 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:00.978073 (+ 82us) client_negotiation.cc:769] Sending connection context
0128 15:03:00.978181 (+ 108us) client_negotiation.cc:240] Negotiation successful
0128 15:03:00.978996 (+ 815us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":673,"spinlock_wait_cycles":2944,"thread_start_us":103,"threads_started":1}
I20260128 15:03:00.979279 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.968310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:49387 (local address 127.13.88.189:38211)
0128 15:03:00.968518 (+ 208us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.968522 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.968538 (+ 16us) server_negotiation.cc:408] Connection header received
0128 15:03:00.968779 (+ 241us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.968783 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.968853 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.968903 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:00.970326 (+ 1423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.971091 (+ 765us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.971962 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.972140 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.975656 (+ 3516us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:00.975677 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:00.975680 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:00.975711 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:00.977584 (+ 1873us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.977976 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.977979 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.977981 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.978027 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.978474 (+ 447us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.978477 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.978479 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.978634 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:00.978712 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.979029 (+ 317us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.979144 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260128 15:03:00.979619 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.968032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.189:38211 (local address 127.13.88.131:49387)
0128 15:03:00.968301 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:00.968313 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:00.968597 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:00.969070 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:00.969077 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:00.969491 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:00.970164 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.970179 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.971225 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.971228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:00.971813 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:00.971821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.972258 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.972710 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:00.972726 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:00.975528 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:00.977692 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.977695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.977697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.977875 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.978325 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:00.978328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:00.978330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:00.978380 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:00.978749 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:00.978752 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:00.978822 (+ 70us) client_negotiation.cc:769] Sending connection context
0128 15:03:00.978910 (+ 88us) client_negotiation.cc:240] Negotiation successful
0128 15:03:00.979460 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":421,"spinlock_wait_cycles":3328,"thread_start_us":101,"threads_started":1}
I20260128 15:03:00.980315 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.966852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:52677 (local address 127.13.88.190:41873)
0128 15:03:00.966997 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.967001 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.967665 (+ 664us) server_negotiation.cc:408] Connection header received
0128 15:03:00.968763 (+ 1098us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.968767 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.968827 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.968901 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:00.970335 (+ 1434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.971002 (+ 667us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.971627 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.971801 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.974304 (+ 2503us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:00.974324 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:00.974328 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:00.974365 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:00.976199 (+ 1834us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.976795 (+ 596us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.976799 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.976801 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.976853 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.977191 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.977195 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.977197 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.977387 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:00.977476 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.980052 (+ 2576us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.980179 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260128 15:03:00.980379 21693 heartbeater.cc:344] Connected to a master server at 127.13.88.188:44165
I20260128 15:03:00.980665 21693 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:00.981268 21693 heartbeater.cc:507] Master 127.13.88.188:44165 requested a full tablet report, sending...
I20260128 15:03:00.980369 21695 heartbeater.cc:344] Connected to a master server at 127.13.88.189:38211
I20260128 15:03:00.982007 21695 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:00.982228 21695 heartbeater.cc:507] Master 127.13.88.189:38211 requested a full tablet report, sending...
I20260128 15:03:00.982373 21224 ts_manager.cc:194] Registered new tserver with Master: fc94066dcfc943bdb867b2d77a4b8efe (127.13.88.131:45859)
I20260128 15:03:00.982873 21694 heartbeater.cc:344] Connected to a master server at 127.13.88.190:41873
I20260128 15:03:00.982939 21694 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:00.983106 21694 heartbeater.cc:507] Master 127.13.88.190:41873 requested a full tablet report, sending...
I20260128 15:03:00.983080 21040 ts_manager.cc:194] Registered new tserver with Master: fc94066dcfc943bdb867b2d77a4b8efe (127.13.88.131:45859)
I20260128 15:03:00.983695 20428 ts_manager.cc:194] Re-registered known tserver with Master: fc94066dcfc943bdb867b2d77a4b8efe (127.13.88.131:45859)
I20260128 15:03:00.984012 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.974538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.13.88.188:44165)
0128 15:03:00.975085 (+ 547us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.975089 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.975101 (+ 12us) server_negotiation.cc:408] Connection header received
0128 15:03:00.975136 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.975139 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.975187 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.975266 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:00.976402 (+ 1136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.977168 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.978151 (+ 983us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.978323 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.980510 (+ 2187us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:00.980528 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:00.980531 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:00.980557 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:00.982238 (+ 1681us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.982692 (+ 454us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.982695 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.982697 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.982744 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.982976 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.982980 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.982981 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.983139 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:00.983257 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.983793 (+ 536us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.983905 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":67,"threads_started":1}
I20260128 15:03:00.984341 20428 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.88.131@KRBTEST.COM'} at 127.13.88.131:52677
I20260128 15:03:00.989135 21707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.974760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.13.88.190:41873)
0128 15:03:00.979633 (+ 4873us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.979637 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.980023 (+ 386us) server_negotiation.cc:408] Connection header received
0128 15:03:00.980064 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.980067 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.980124 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.980195 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:00.981065 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.981803 (+ 738us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.984604 (+ 2801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.984788 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.986300 (+ 1512us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:00.986314 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:00.986317 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:00.986339 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:00.987795 (+ 1456us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.988164 (+ 369us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.988167 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.988168 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.988206 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.988403 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.988406 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.988408 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.988584 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:00.988707 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.988872 (+ 165us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.988976 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1377,"thread_start_us":63,"threads_started":1}
I20260128 15:03:00.989256 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:00.974889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57766 (local address 127.13.88.189:38211)
0128 15:03:00.979692 (+ 4803us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:00.979696 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:00.981044 (+ 1348us) server_negotiation.cc:408] Connection header received
0128 15:03:00.981080 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:00.981083 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:00.981125 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:00.981197 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:00.983225 (+ 2028us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.983915 (+ 690us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:00.985097 (+ 1182us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:00.985245 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:00.986285 (+ 1040us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:00.986302 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:00.986305 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:00.986330 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:00.987994 (+ 1664us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.988309 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.988312 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.988313 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.988358 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:00.988564 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:00.988568 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:00.988570 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:00.988717 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:00.988821 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:00.989014 (+ 193us) server_negotiation.cc:300] Negotiation successful
0128 15:03:00.989115 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141,"thread_start_us":66,"threads_started":1}
I20260128 15:03:00.991389 20428 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 15:03:00.993618 20428 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:03:01.008610 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:01.004905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:33663 (local address 127.0.0.1:55156)
0128 15:03:01.005461 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:01.005473 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:01.005555 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:01.006124 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:01.006127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:01.006148 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:01.006425 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:01.006434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.007474 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:01.007477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:01.008199 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:01.008207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.008299 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:01.008329 (+ 30us) client_negotiation.cc:769] Sending connection context
0128 15:03:01.008377 (+ 48us) client_negotiation.cc:240] Negotiation successful
0128 15:03:01.008462 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"spinlock_wait_cycles":5888,"thread_start_us":109,"threads_started":1}
I20260128 15:03:01.009253 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:01.005044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.13.88.129:33663)
0128 15:03:01.005373 (+ 329us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:01.005380 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:01.005546 (+ 166us) server_negotiation.cc:408] Connection header received
0128 15:03:01.005752 (+ 206us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:01.005759 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:01.005962 (+ 203us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:01.006548 (+ 586us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:01.006674 (+ 126us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.007366 (+ 692us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:01.008330 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.008883 (+ 553us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:01.009006 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:01.009077 (+ 71us) server_negotiation.cc:300] Negotiation successful
0128 15:03:01.009151 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":121,"threads_started":1}
I20260128 15:03:01.009382 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:01.005215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:41581 (local address 127.0.0.1:49036)
0128 15:03:01.005772 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:01.005784 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:01.005860 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:01.006408 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:01.006411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:01.006429 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:01.006650 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:01.006656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.007535 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:01.007539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:01.009039 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:01.009047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.009143 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:01.009162 (+ 19us) client_negotiation.cc:769] Sending connection context
0128 15:03:01.009206 (+ 44us) client_negotiation.cc:240] Negotiation successful
0128 15:03:01.009259 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":91,"threads_started":1}
I20260128 15:03:01.009939 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:01.005396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49036 (local address 127.13.88.130:41581)
0128 15:03:01.005723 (+ 327us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:01.005729 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:01.006024 (+ 295us) server_negotiation.cc:408] Connection header received
0128 15:03:01.006117 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:01.006122 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:01.006276 (+ 154us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:01.006414 (+ 138us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:01.006755 (+ 341us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.007436 (+ 681us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:01.009164 (+ 1728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.009655 (+ 491us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:01.009734 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:01.009781 (+ 47us) server_negotiation.cc:300] Negotiation successful
0128 15:03:01.009841 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":94,"threads_started":1}
I20260128 15:03:01.011193 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:01.004587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:45859 (local address 127.0.0.1:53180)
0128 15:03:01.005439 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:01.005453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:01.005547 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:01.007484 (+ 1937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:01.007487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:01.007508 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:01.007737 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:01.007743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.010013 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:01.010017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:01.010844 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:01.010853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.010962 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:01.010978 (+ 16us) client_negotiation.cc:769] Sending connection context
0128 15:03:01.011021 (+ 43us) client_negotiation.cc:240] Negotiation successful
0128 15:03:01.011082 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":106,"threads_started":1}
I20260128 15:03:01.011719 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:01.004748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53180 (local address 127.13.88.131:45859)
0128 15:03:01.006095 (+ 1347us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:01.006101 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:01.006121 (+ 20us) server_negotiation.cc:408] Connection header received
0128 15:03:01.006198 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:01.006203 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:01.006375 (+ 172us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:01.006505 (+ 130us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:01.007857 (+ 1352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.008893 (+ 1036us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:01.010963 (+ 2070us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.011361 (+ 398us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:01.011461 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:01.011522 (+ 61us) server_negotiation.cc:300] Negotiation successful
0128 15:03:01.011600 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":84,"threads_started":1}
I20260128 15:03:01.012182 21355 tablet_service.cc:1505] Processing CreateTablet for tablet fd059cad1b124b5e88a0bf425bb6893a (DEFAULT_TABLE table=test-table [id=f0dbea999b7f4e499454a8df9845fbea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:01.013202 21355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd059cad1b124b5e88a0bf425bb6893a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:01.014323 21627 tablet_service.cc:1505] Processing CreateTablet for tablet fd059cad1b124b5e88a0bf425bb6893a (DEFAULT_TABLE table=test-table [id=f0dbea999b7f4e499454a8df9845fbea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:01.015246 21627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd059cad1b124b5e88a0bf425bb6893a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:01.016150 21491 tablet_service.cc:1505] Processing CreateTablet for tablet fd059cad1b124b5e88a0bf425bb6893a (DEFAULT_TABLE table=test-table [id=f0dbea999b7f4e499454a8df9845fbea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:01.017035 21491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd059cad1b124b5e88a0bf425bb6893a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:01.022138 21724 tablet_bootstrap.cc:492] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe: Bootstrap starting.
I20260128 15:03:01.023593 21725 tablet_bootstrap.cc:492] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e: Bootstrap starting.
I20260128 15:03:01.024705 21724 tablet_bootstrap.cc:654] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:01.025447 21725 tablet_bootstrap.cc:654] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:01.025540 21724 log.cc:826] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe: Log is configured to *not* fsync() on all Append() calls
I20260128 15:03:01.025661 21726 tablet_bootstrap.cc:492] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21: Bootstrap starting.
I20260128 15:03:01.026331 21725 log.cc:826] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e: Log is configured to *not* fsync() on all Append() calls
I20260128 15:03:01.027662 21726 tablet_bootstrap.cc:654] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:01.027697 21724 tablet_bootstrap.cc:492] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe: No bootstrap required, opened a new log
I20260128 15:03:01.027834 21724 ts_tablet_manager.cc:1403] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260128 15:03:01.028481 21726 log.cc:826] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21: Log is configured to *not* fsync() on all Append() calls
I20260128 15:03:01.028532 21725 tablet_bootstrap.cc:492] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e: No bootstrap required, opened a new log
I20260128 15:03:01.028689 21725 ts_tablet_manager.cc:1403] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260128 15:03:01.030139 21726 tablet_bootstrap.cc:492] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21: No bootstrap required, opened a new log
I20260128 15:03:01.030323 21726 ts_tablet_manager.cc:1403] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260128 15:03:01.030308 21724 raft_consensus.cc:359] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } }
I20260128 15:03:01.030555 21724 raft_consensus.cc:385] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:01.030627 21724 raft_consensus.cc:740] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc94066dcfc943bdb867b2d77a4b8efe, State: Initialized, Role: FOLLOWER
I20260128 15:03:01.031037 21724 consensus_queue.cc:260] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } }
I20260128 15:03:01.031510 21725 raft_consensus.cc:359] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } }
I20260128 15:03:01.031795 21725 raft_consensus.cc:385] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:01.031857 21724 ts_tablet_manager.cc:1434] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260128 15:03:01.031881 21725 raft_consensus.cc:740] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e25f3cc9ae44aba196ed92a3adfb0e, State: Initialized, Role: FOLLOWER
I20260128 15:03:01.032061 21694 heartbeater.cc:499] Master 127.13.88.190:41873 was elected leader, sending a full tablet report...
I20260128 15:03:01.032454 21725 consensus_queue.cc:260] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } }
I20260128 15:03:01.033324 21725 ts_tablet_manager.cc:1434] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260128 15:03:01.034971 21558 heartbeater.cc:499] Master 127.13.88.190:41873 was elected leader, sending a full tablet report...
I20260128 15:03:01.036111 21726 raft_consensus.cc:359] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } }
I20260128 15:03:01.036392 21726 raft_consensus.cc:385] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:01.036511 21726 raft_consensus.cc:740] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc2239548eb481580722ea108a90e21, State: Initialized, Role: FOLLOWER
I20260128 15:03:01.036989 21726 consensus_queue.cc:260] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } }
I20260128 15:03:01.037600 21422 heartbeater.cc:499] Master 127.13.88.190:41873 was elected leader, sending a full tablet report...
I20260128 15:03:01.037750 21726 ts_tablet_manager.cc:1434] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260128 15:03:01.104136 21424 tablet.cc:2378] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 15:03:01.117105 21731 raft_consensus.cc:493] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:03:01.117331 21731 raft_consensus.cc:515] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } }
I20260128 15:03:01.118541 21731 leader_election.cc:290] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc94066dcfc943bdb867b2d77a4b8efe (127.13.88.131:45859), 3fc2239548eb481580722ea108a90e21 (127.13.88.129:33663)
I20260128 15:03:01.121440 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:01.118798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:33663 (local address 127.13.88.130:38759)
0128 15:03:01.118928 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:01.118943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:01.119040 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:01.119342 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:01.119345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:01.119367 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:01.119624 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:01.119631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.120477 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:01.120482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:01.121146 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:01.121155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.121243 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:01.121261 (+ 18us) client_negotiation.cc:769] Sending connection context
0128 15:03:01.121294 (+ 33us) client_negotiation.cc:240] Negotiation successful
0128 15:03:01.121341 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260128 15:03:01.121752 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:01.118812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:45859 (local address 127.13.88.130:34841)
0128 15:03:01.119238 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:01.119251 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:01.119332 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:01.119602 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:01.119604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:01.119617 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:01.119820 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:01.119827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.120683 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:01.120687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:01.121392 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:01.121401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.121515 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:01.121534 (+ 19us) client_negotiation.cc:769] Sending connection context
0128 15:03:01.121579 (+ 45us) client_negotiation.cc:240] Negotiation successful
0128 15:03:01.121630 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":303}
I20260128 15:03:01.122025 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:01.118995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:38759 (local address 127.13.88.129:33663)
0128 15:03:01.119110 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:01.119114 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:01.119127 (+ 13us) server_negotiation.cc:408] Connection header received
0128 15:03:01.119181 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:01.119185 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:01.119235 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:01.119324 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:01.119738 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.120290 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:01.121266 (+ 976us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.121752 (+ 486us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:01.121791 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:01.121842 (+ 51us) server_negotiation.cc:300] Negotiation successful
0128 15:03:01.121908 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260128 15:03:01.122222 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:01.118863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:34841 (local address 127.13.88.131:45859)
0128 15:03:01.118989 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:01.118993 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:01.119334 (+ 341us) server_negotiation.cc:408] Connection header received
0128 15:03:01.119437 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:01.119440 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:01.119519 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:01.119589 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:01.119922 (+ 333us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.120555 (+ 633us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:01.121520 (+ 965us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.121991 (+ 471us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:01.122029 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:01.122069 (+ 40us) server_negotiation.cc:300] Negotiation successful
0128 15:03:01.122117 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260128 15:03:01.122563 21375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd059cad1b124b5e88a0bf425bb6893a" candidate_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc2239548eb481580722ea108a90e21" is_pre_election: true
I20260128 15:03:01.122773 21647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd059cad1b124b5e88a0bf425bb6893a" candidate_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" is_pre_election: true
I20260128 15:03:01.122836 21375 raft_consensus.cc:2468] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e25f3cc9ae44aba196ed92a3adfb0e in term 0.
I20260128 15:03:01.122984 21647 raft_consensus.cc:2468] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e25f3cc9ae44aba196ed92a3adfb0e in term 0.
I20260128 15:03:01.123381 21445 leader_election.cc:304] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3fc2239548eb481580722ea108a90e21, e5e25f3cc9ae44aba196ed92a3adfb0e, fc94066dcfc943bdb867b2d77a4b8efe; no voters:
I20260128 15:03:01.123637 21731 raft_consensus.cc:2804] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:03:01.123713 21731 raft_consensus.cc:493] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:03:01.123781 21731 raft_consensus.cc:3060] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:01.124996 21731 raft_consensus.cc:515] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } }
I20260128 15:03:01.125384 21731 leader_election.cc:290] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [CANDIDATE]: Term 1 election: Requested vote from peers fc94066dcfc943bdb867b2d77a4b8efe (127.13.88.131:45859), 3fc2239548eb481580722ea108a90e21 (127.13.88.129:33663)
I20260128 15:03:01.125686 21647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd059cad1b124b5e88a0bf425bb6893a" candidate_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc94066dcfc943bdb867b2d77a4b8efe"
I20260128 15:03:01.125732 21375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd059cad1b124b5e88a0bf425bb6893a" candidate_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc2239548eb481580722ea108a90e21"
I20260128 15:03:01.125788 21647 raft_consensus.cc:3060] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:01.125852 21375 raft_consensus.cc:3060] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:01.126681 21647 raft_consensus.cc:2468] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e25f3cc9ae44aba196ed92a3adfb0e in term 1.
I20260128 15:03:01.126700 21375 raft_consensus.cc:2468] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e25f3cc9ae44aba196ed92a3adfb0e in term 1.
I20260128 15:03:01.126966 21446 leader_election.cc:304] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e5e25f3cc9ae44aba196ed92a3adfb0e, fc94066dcfc943bdb867b2d77a4b8efe; no voters:
I20260128 15:03:01.127121 21731 raft_consensus.cc:2804] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:03:01.127318 21731 raft_consensus.cc:697] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [term 1 LEADER]: Becoming Leader. State: Replica: e5e25f3cc9ae44aba196ed92a3adfb0e, State: Running, Role: LEADER
I20260128 15:03:01.127628 21731 consensus_queue.cc:237] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } }
I20260128 15:03:01.130854 20428 catalog_manager.cc:5654] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e reported cstate change: term changed from 0 to 1, leader changed from <none> to e5e25f3cc9ae44aba196ed92a3adfb0e (127.13.88.130). New cstate: current_term: 1 leader_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } health_report { overall_health: HEALTHY } } }
I20260128 15:03:01.161202 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:01.158268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49038 (local address 127.13.88.130:41581)
0128 15:03:01.158405 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:01.158410 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:01.158424 (+ 14us) server_negotiation.cc:408] Connection header received
0128 15:03:01.158531 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:01.158534 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:01.158588 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:01.158667 (+ 79us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:03:01.159002 (+ 335us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.159492 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:01.160116 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.160296 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:01.160394 (+ 98us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:03:01.160848 (+ 454us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:03:01.160958 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:01.161013 (+ 55us) server_negotiation.cc:300] Negotiation successful
0128 15:03:01.161064 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260128 15:03:01.166507 21375 raft_consensus.cc:1275] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21 [term 1 FOLLOWER]: Refusing update from remote peer e5e25f3cc9ae44aba196ed92a3adfb0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:03:01.166507 21647 raft_consensus.cc:1275] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe [term 1 FOLLOWER]: Refusing update from remote peer e5e25f3cc9ae44aba196ed92a3adfb0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:03:01.167366 21733 consensus_queue.cc:1048] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:03:01.167521 21731 consensus_queue.cc:1048] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:03:01.175326 21737 mvcc.cc:204] Tried to move back new op lower bound from 7248333132451278848 to 7248333132301344768. Current Snapshot: MvccSnapshot[applied={T|T < 7248333132451278848}]
I20260128 15:03:01.178228 21738 mvcc.cc:204] Tried to move back new op lower bound from 7248333132451278848 to 7248333132301344768. Current Snapshot: MvccSnapshot[applied={T|T < 7248333132451278848}]
I20260128 15:03:01.183931 20428 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:03:01.184085 20428 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:03:01.187987 20428 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 514b33d5af9e48e49897a8adce27696b: Sending DeleteTablet for 3 replicas of tablet fd059cad1b124b5e88a0bf425bb6893a
I20260128 15:03:01.188812 21627 tablet_service.cc:1552] Processing DeleteTablet for tablet fd059cad1b124b5e88a0bf425bb6893a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:01 UTC) from {username='oryx', principal='oryx/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:53180
I20260128 15:03:01.189102 21491 tablet_service.cc:1552] Processing DeleteTablet for tablet fd059cad1b124b5e88a0bf425bb6893a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:01 UTC) from {username='oryx', principal='oryx/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:49036
I20260128 15:03:01.189278 21355 tablet_service.cc:1552] Processing DeleteTablet for tablet fd059cad1b124b5e88a0bf425bb6893a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:01 UTC) from {username='oryx', principal='oryx/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:55156
I20260128 15:03:01.189510 21746 tablet_replica.cc:333] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e: stopping tablet replica
I20260128 15:03:01.189752 21746 raft_consensus.cc:2243] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [term 1 LEADER]: Raft consensus shutting down.
I20260128 15:03:01.190151 21746 raft_consensus.cc:2272] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:01.190171 21747 tablet_replica.cc:333] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21: stopping tablet replica
I20260128 15:03:01.190447 21747 raft_consensus.cc:2243] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:03:01.190791 21747 raft_consensus.cc:2272] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:01.191256 21745 tablet_replica.cc:333] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe: stopping tablet replica
I20260128 15:03:01.191838 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 20397
I20260128 15:03:01.192277 21747 ts_tablet_manager.cc:1916] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:01.192945 21746 ts_tablet_manager.cc:1916] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:01.194706 21747 ts_tablet_manager.cc:1929] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:03:01.194995 21745 raft_consensus.cc:2243] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:03:01.195320 21745 raft_consensus.cc:2272] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:01.195367 21746 ts_tablet_manager.cc:1929] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:03:01.195456 21746 log.cc:1199] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/wal/wals/fd059cad1b124b5e88a0bf425bb6893a
I20260128 15:03:01.195554 21747 log.cc:1199] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-0/wal/wals/fd059cad1b124b5e88a0bf425bb6893a
I20260128 15:03:01.195870 21747 ts_tablet_manager.cc:1950] T fd059cad1b124b5e88a0bf425bb6893a P 3fc2239548eb481580722ea108a90e21: Deleting consensus metadata
W20260128 15:03:01.197702 21309 connection.cc:537] server connection from 127.0.0.1:55156 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260128 15:03:01.198573 21746 ts_tablet_manager.cc:1950] T fd059cad1b124b5e88a0bf425bb6893a P e5e25f3cc9ae44aba196ed92a3adfb0e: Deleting consensus metadata
I20260128 15:03:01.199280 21745 ts_tablet_manager.cc:1916] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260128 15:03:01.199468 21445 connection.cc:441] server connection from 127.0.0.1:49036 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49036 (request call id 1) could send its response
I20260128 15:03:01.201699 21745 ts_tablet_manager.cc:1929] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:03:01.201859 21745 log.cc:1199] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/wal/wals/fd059cad1b124b5e88a0bf425bb6893a
I20260128 15:03:01.202181 21745 ts_tablet_manager.cc:1950] T fd059cad1b124b5e88a0bf425bb6893a P fc94066dcfc943bdb867b2d77a4b8efe: Deleting consensus metadata
W20260128 15:03:01.203428 21580 connection.cc:441] server connection from 127.0.0.1:53180 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53180 (request call id 1) could send its response
I20260128 15:03:01.207762 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:01.200218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34132 (local address 127.13.88.188:44165)
0128 15:03:01.200338 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:01.200342 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:01.200639 (+ 297us) server_negotiation.cc:408] Connection header received
0128 15:03:01.200756 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:01.200759 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:01.200803 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:01.200862 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:01.201784 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.202388 (+ 604us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:01.203003 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.203201 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:01.204259 (+ 1058us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:01.204276 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:01.204278 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:01.204303 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:01.206037 (+ 1734us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:01.206432 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:01.206435 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:01.206437 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:01.206479 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:01.206662 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:01.206665 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:01.206667 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:01.206821 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:01.206930 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:01.207494 (+ 564us) server_negotiation.cc:300] Negotiation successful
0128 15:03:01.207610 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260128 15:03:01.212620 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:01.205209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57776 (local address 127.13.88.189:38211)
0128 15:03:01.205336 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:01.205340 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:01.205524 (+ 184us) server_negotiation.cc:408] Connection header received
0128 15:03:01.205637 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:01.205641 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:01.205698 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:01.205778 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:01.206845 (+ 1067us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.207588 (+ 743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:01.208593 (+ 1005us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:01.208746 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:01.209857 (+ 1111us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:01.209873 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:01.209876 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:01.209899 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:01.211209 (+ 1310us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:01.211607 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:01.211611 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:01.211612 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:01.211661 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:01.211854 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:01.211856 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:01.211857 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:01.211980 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:01.212087 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:01.212268 (+ 181us) server_negotiation.cc:300] Negotiation successful
0128 15:03:01.212366 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260128 15:03:02.179987 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:02.179546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.129:33211)
0128 15:03:02.179787 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:02.179873 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":74,"threads_started":1}
W20260128 15:03:02.180395 21422 heartbeater.cc:646] Failed to heartbeat to 127.13.88.190:41873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
I20260128 15:03:02.181319 21755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:02.180919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.131:60377)
0128 15:03:02.181154 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:02.181227 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":73,"threads_started":1}
I20260128 15:03:02.181519 21756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:02.181011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.130:49031)
0128 15:03:02.181253 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:02.181312 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":68,"threads_started":1}
W20260128 15:03:02.181545 21694 heartbeater.cc:646] Failed to heartbeat to 127.13.88.190:41873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
W20260128 15:03:02.181785 21558 heartbeater.cc:646] Failed to heartbeat to 127.13.88.190:41873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
I20260128 15:03:02.688938 21757 raft_consensus.cc:493] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 514b33d5af9e48e49897a8adce27696b)
I20260128 15:03:02.689116 21757 raft_consensus.cc:515] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } }
I20260128 15:03:02.689817 21758 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 514b33d5af9e48e49897a8adce27696b)
I20260128 15:03:02.689957 21758 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } }
I20260128 15:03:02.690222 21757 leader_election.cc:290] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 514b33d5af9e48e49897a8adce27696b (127.13.88.190:41873), 1c36b3bafb1547efb54a35126ed2c62c (127.13.88.189:38211)
I20260128 15:03:02.690893 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:02.690377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:42620)
0128 15:03:02.690721 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:02.690792 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":57,"threads_started":1}
I20260128 15:03:02.690919 21050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c103d7f4e85045059b10b02c5788eaba" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1c36b3bafb1547efb54a35126ed2c62c" is_pre_election: true
I20260128 15:03:02.691092 21050 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c103d7f4e85045059b10b02c5788eaba in term 1.
W20260128 15:03:02.691066 21211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
I20260128 15:03:02.691170 21234 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c36b3bafb1547efb54a35126ed2c62c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c103d7f4e85045059b10b02c5788eaba" is_pre_election: true
I20260128 15:03:02.691325 21234 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c36b3bafb1547efb54a35126ed2c62c in term 1.
I20260128 15:03:02.691406 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:02.690912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:42626)
0128 15:03:02.691242 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:02.691305 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":57,"threads_started":1}
I20260128 15:03:02.691427 21758 leader_election.cc:290] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 514b33d5af9e48e49897a8adce27696b (127.13.88.190:41873), c103d7f4e85045059b10b02c5788eaba (127.13.88.188:44165)
I20260128 15:03:02.691601 21211 leader_election.cc:304] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c36b3bafb1547efb54a35126ed2c62c, c103d7f4e85045059b10b02c5788eaba; no voters:
I20260128 15:03:02.691672 21024 leader_election.cc:304] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c36b3bafb1547efb54a35126ed2c62c, c103d7f4e85045059b10b02c5788eaba; no voters:
W20260128 15:03:02.691708 21027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
I20260128 15:03:02.691859 21758 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260128 15:03:02.691916 21758 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 514b33d5af9e48e49897a8adce27696b)
I20260128 15:03:02.691895 21757 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260128 15:03:02.691956 21758 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 1 FOLLOWER]: Advancing to term 2
I20260128 15:03:02.691956 21757 raft_consensus.cc:493] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 1 FOLLOWER]: Starting leader election (detected failure of leader 514b33d5af9e48e49897a8adce27696b)
I20260128 15:03:02.691999 21757 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 1 FOLLOWER]: Advancing to term 2
I20260128 15:03:02.692127 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:02.691878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:42640)
0128 15:03:02.692022 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:02.692066 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260128 15:03:02.692238 21211 leader_election.cc:336] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 514b33d5af9e48e49897a8adce27696b (127.13.88.190:41873): Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
I20260128 15:03:02.692759 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:02.692539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:42656)
0128 15:03:02.692652 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:02.692699 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260128 15:03:02.692853 21027 leader_election.cc:336] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 514b33d5af9e48e49897a8adce27696b (127.13.88.190:41873): Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
I20260128 15:03:02.693233 21757 raft_consensus.cc:515] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } }
I20260128 15:03:02.693334 21758 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } }
I20260128 15:03:02.693540 21757 leader_election.cc:290] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [CANDIDATE]: Term 2 election: Requested vote from peers 514b33d5af9e48e49897a8adce27696b (127.13.88.190:41873), 1c36b3bafb1547efb54a35126ed2c62c (127.13.88.189:38211)
I20260128 15:03:02.693843 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:02.693632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:42662)
0128 15:03:02.693741 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:02.693785 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260128 15:03:02.693917 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:02.693712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:42678)
0128 15:03:02.693809 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:02.693853 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260128 15:03:02.693964 21050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c103d7f4e85045059b10b02c5788eaba" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1c36b3bafb1547efb54a35126ed2c62c"
I20260128 15:03:02.694103 21050 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c103d7f4e85045059b10b02c5788eaba in current term 2: Already voted for candidate 1c36b3bafb1547efb54a35126ed2c62c in this term.
I20260128 15:03:02.694139 21234 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c36b3bafb1547efb54a35126ed2c62c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c103d7f4e85045059b10b02c5788eaba"
I20260128 15:03:02.694284 21234 raft_consensus.cc:2393] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c36b3bafb1547efb54a35126ed2c62c in current term 2: Already voted for candidate c103d7f4e85045059b10b02c5788eaba in this term.
I20260128 15:03:02.694587 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:02.694272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:42690)
0128 15:03:02.694468 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:02.694515 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":54,"threads_started":1}
I20260128 15:03:02.694589 21758 leader_election.cc:290] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [CANDIDATE]: Term 2 election: Requested vote from peers 514b33d5af9e48e49897a8adce27696b (127.13.88.190:41873), c103d7f4e85045059b10b02c5788eaba (127.13.88.188:44165)
W20260128 15:03:02.694746 21027 leader_election.cc:336] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 514b33d5af9e48e49897a8adce27696b (127.13.88.190:41873): Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
I20260128 15:03:02.694813 21027 leader_election.cc:304] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c36b3bafb1547efb54a35126ed2c62c; no voters: 514b33d5af9e48e49897a8adce27696b, c103d7f4e85045059b10b02c5788eaba
I20260128 15:03:02.694933 21758 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260128 15:03:02.695052 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:02.694828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:42700)
0128 15:03:02.694955 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:02.694996 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260128 15:03:02.695151 21211 leader_election.cc:336] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 514b33d5af9e48e49897a8adce27696b (127.13.88.190:41873): Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
I20260128 15:03:02.695211 21211 leader_election.cc:304] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c103d7f4e85045059b10b02c5788eaba; no voters: 1c36b3bafb1547efb54a35126ed2c62c, 514b33d5af9e48e49897a8adce27696b
I20260128 15:03:02.695317 21757 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260128 15:03:03.181558 21764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:03.180982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.129:50681)
0128 15:03:03.181367 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:03.181445 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":97,"threads_started":1}
I20260128 15:03:03.182435 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:03.182004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.131:42989)
0128 15:03:03.182279 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:03.182344 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":66,"threads_started":1}
I20260128 15:03:03.182760 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:03.182229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.130:38133)
0128 15:03:03.182550 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:03.182612 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260128 15:03:04.182829 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.182308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.129:51429)
0128 15:03:04.182643 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.182715 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":98,"threads_started":1}
I20260128 15:03:04.183754 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.183126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.131:35813)
0128 15:03:04.183604 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.183662 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":61,"threads_started":1}
I20260128 15:03:04.183882 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.183544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.130:43581)
0128 15:03:04.183740 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.183791 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20260128 15:03:04.223171 21770 raft_consensus.cc:493] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:03:04.223313 21770 raft_consensus.cc:515] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } }
I20260128 15:03:04.223727 21770 leader_election.cc:290] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 514b33d5af9e48e49897a8adce27696b (127.13.88.190:41873), 1c36b3bafb1547efb54a35126ed2c62c (127.13.88.189:38211)
I20260128 15:03:04.224336 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.223953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:42938)
0128 15:03:04.224168 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.224237 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":64,"threads_started":1}
I20260128 15:03:04.224555 21050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c103d7f4e85045059b10b02c5788eaba" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1c36b3bafb1547efb54a35126ed2c62c" is_pre_election: true
I20260128 15:03:04.224713 21050 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c103d7f4e85045059b10b02c5788eaba in term 2.
I20260128 15:03:04.225808 21211 leader_election.cc:304] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c36b3bafb1547efb54a35126ed2c62c, c103d7f4e85045059b10b02c5788eaba; no voters:
I20260128 15:03:04.225891 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.225666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:42950)
0128 15:03:04.225780 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.225828 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260128 15:03:04.225965 21211 leader_election.cc:336] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 514b33d5af9e48e49897a8adce27696b (127.13.88.190:41873): Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
I20260128 15:03:04.226001 21770 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260128 15:03:04.226053 21770 raft_consensus.cc:493] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:03:04.226100 21770 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 2 FOLLOWER]: Advancing to term 3
I20260128 15:03:04.226991 21770 raft_consensus.cc:515] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } }
I20260128 15:03:04.227308 21770 leader_election.cc:290] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [CANDIDATE]: Term 3 election: Requested vote from peers 514b33d5af9e48e49897a8adce27696b (127.13.88.190:41873), 1c36b3bafb1547efb54a35126ed2c62c (127.13.88.189:38211)
I20260128 15:03:04.227620 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.227403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:42952)
0128 15:03:04.227520 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.227565 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260128 15:03:04.227761 21050 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c103d7f4e85045059b10b02c5788eaba" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1c36b3bafb1547efb54a35126ed2c62c"
I20260128 15:03:04.227862 21050 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 2 FOLLOWER]: Advancing to term 3
I20260128 15:03:04.228180 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.228008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:42958)
0128 15:03:04.228094 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.228134 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260128 15:03:04.228266 21211 leader_election.cc:336] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 514b33d5af9e48e49897a8adce27696b (127.13.88.190:41873): Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
I20260128 15:03:04.228751 21050 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c103d7f4e85045059b10b02c5788eaba in term 3.
I20260128 15:03:04.228991 21211 leader_election.cc:304] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c36b3bafb1547efb54a35126ed2c62c, c103d7f4e85045059b10b02c5788eaba; no voters: 514b33d5af9e48e49897a8adce27696b
I20260128 15:03:04.229166 21770 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 3 FOLLOWER]: Leader election won for term 3
I20260128 15:03:04.229351 21770 raft_consensus.cc:697] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [term 3 LEADER]: Becoming Leader. State: Replica: c103d7f4e85045059b10b02c5788eaba, State: Running, Role: LEADER
I20260128 15:03:04.229583 21770 consensus_queue.cc:237] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } }
I20260128 15:03:04.230919 21773 sys_catalog.cc:455] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [sys.catalog]: SysCatalogTable state changed. Reason: New leader c103d7f4e85045059b10b02c5788eaba. Latest consensus state: current_term: 3 leader_uuid: "c103d7f4e85045059b10b02c5788eaba" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } } }
I20260128 15:03:04.231009 21773 sys_catalog.cc:458] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:04.231279 21775 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:03:04.233315 21775 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:03:04.233665 21775 catalog_manager.cc:1269] Loaded cluster ID: 160ee1f718a440ada4dbc6a39dc86fae
I20260128 15:03:04.233708 21775 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:03:04.234197 21775 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:03:04.234549 21775 catalog_manager.cc:6038] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: Loaded TSK: 0
I20260128 15:03:04.235348 21775 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260128 15:03:04.245355 21222 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 15:03:04.247545 21222 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:03:04.250097 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.249907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:42982)
0128 15:03:04.249997 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.250040 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260128 15:03:04.250332 21050 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [term 3 FOLLOWER]: Refusing update from remote peer c103d7f4e85045059b10b02c5788eaba: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260128 15:03:04.250766 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.250586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.0.0.1:42984)
0128 15:03:04.250676 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.250717 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260128 15:03:04.250790 21770 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260128 15:03:04.250948 21211 consensus_peers.cc:597] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba -> Peer 514b33d5af9e48e49897a8adce27696b (127.13.88.190:41873): Couldn't send request to peer 514b33d5af9e48e49897a8adce27696b. Status: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260128 15:03:04.252082 21777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c103d7f4e85045059b10b02c5788eaba. Latest consensus state: current_term: 3 leader_uuid: "c103d7f4e85045059b10b02c5788eaba" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } } }
I20260128 15:03:04.252192 21777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: This master's current role is: FOLLOWER
I20260128 15:03:04.253082 21770 sys_catalog.cc:455] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "c103d7f4e85045059b10b02c5788eaba" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } } }
I20260128 15:03:04.253197 21770 sys_catalog.cc:458] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:04.253571 21777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "c103d7f4e85045059b10b02c5788eaba" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "514b33d5af9e48e49897a8adce27696b" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41873 } } peers { permanent_uuid: "1c36b3bafb1547efb54a35126ed2c62c" member_type: VOTER last_known_addr { host: "127.13.88.189" port: 38211 } attrs { promote: false } } peers { permanent_uuid: "c103d7f4e85045059b10b02c5788eaba" member_type: VOTER last_known_addr { host: "127.13.88.188" port: 44165 } attrs { promote: false } } }
I20260128 15:03:04.253669 21777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c36b3bafb1547efb54a35126ed2c62c [sys.catalog]: This master's current role is: FOLLOWER
I20260128 15:03:04.261616 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.258137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:41581 (local address 127.0.0.1:49454)
0128 15:03:04.258284 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.258298 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:04.258383 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:04.259104 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:04.259116 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:04.259133 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:04.259382 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:04.259390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.260335 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:04.260339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:04.261245 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:04.261254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.261358 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:04.261376 (+ 18us) client_negotiation.cc:769] Sending connection context
0128 15:03:04.261424 (+ 48us) client_negotiation.cc:240] Negotiation successful
0128 15:03:04.261482 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260128 15:03:04.261617 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.258467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:33663 (local address 127.0.0.1:40272)
0128 15:03:04.258804 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.258817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:04.258892 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:04.259226 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:04.259229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:04.259244 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:04.259447 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:04.259452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.260280 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:04.260284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:04.261245 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:04.261254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.261358 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:04.261376 (+ 18us) client_negotiation.cc:769] Sending connection context
0128 15:03:04.261433 (+ 57us) client_negotiation.cc:240] Negotiation successful
0128 15:03:04.261481 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":72,"threads_started":1}
I20260128 15:03:04.261858 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.258792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:45859 (local address 127.0.0.1:36650)
0128 15:03:04.259158 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.259171 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:04.259238 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:04.259576 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:04.259578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:04.259596 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:04.259805 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:04.259811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.260766 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:04.260769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:04.261522 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:04.261532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.261631 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:04.261646 (+ 15us) client_negotiation.cc:769] Sending connection context
0128 15:03:04.261690 (+ 44us) client_negotiation.cc:240] Negotiation successful
0128 15:03:04.261744 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":81,"threads_started":1}
I20260128 15:03:04.262073 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.258214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49454 (local address 127.13.88.130:41581)
0128 15:03:04.258880 (+ 666us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:04.258885 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:04.258899 (+ 14us) server_negotiation.cc:408] Connection header received
0128 15:03:04.258948 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:04.258951 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:04.258996 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:04.259486 (+ 490us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:04.259597 (+ 111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.260243 (+ 646us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:04.261366 (+ 1123us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.261814 (+ 448us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:04.261847 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:04.261895 (+ 48us) server_negotiation.cc:300] Negotiation successful
0128 15:03:04.261958 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":72,"threads_started":1}
I20260128 15:03:04.262521 21491 tablet_service.cc:1505] Processing CreateTablet for tablet 0eb31e7e746b493e839dd10943182c97 (DEFAULT_TABLE table=test-table [id=574387a8993f41cb89ccc037c673ea7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:04.262735 21787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.259123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36650 (local address 127.13.88.131:45859)
0128 15:03:04.259365 (+ 242us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:04.259369 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:04.259383 (+ 14us) server_negotiation.cc:408] Connection header received
0128 15:03:04.259430 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:04.259433 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:04.259484 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:04.259570 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:04.259907 (+ 337us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.260653 (+ 746us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:04.262024 (+ 1371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.262478 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:04.262512 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:04.262559 (+ 47us) server_negotiation.cc:300] Negotiation successful
0128 15:03:04.262611 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
I20260128 15:03:04.262771 21491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eb31e7e746b493e839dd10943182c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:04.263123 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.258570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.13.88.129:33663)
0128 15:03:04.258907 (+ 337us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:04.258911 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:04.258928 (+ 17us) server_negotiation.cc:408] Connection header received
0128 15:03:04.258977 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:04.258980 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:04.259030 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:04.259109 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:04.259555 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.260186 (+ 631us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:04.262367 (+ 2181us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.262820 (+ 453us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:04.262856 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:04.262903 (+ 47us) server_negotiation.cc:300] Negotiation successful
0128 15:03:04.262960 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":73,"threads_started":1}
I20260128 15:03:04.263216 21627 tablet_service.cc:1505] Processing CreateTablet for tablet 0eb31e7e746b493e839dd10943182c97 (DEFAULT_TABLE table=test-table [id=574387a8993f41cb89ccc037c673ea7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:04.263484 21627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eb31e7e746b493e839dd10943182c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:04.264679 21355 tablet_service.cc:1505] Processing CreateTablet for tablet 0eb31e7e746b493e839dd10943182c97 (DEFAULT_TABLE table=test-table [id=574387a8993f41cb89ccc037c673ea7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:04.264922 21355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eb31e7e746b493e839dd10943182c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:04.265002 21788 tablet_bootstrap.cc:492] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e: Bootstrap starting.
I20260128 15:03:04.265364 21789 tablet_bootstrap.cc:492] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe: Bootstrap starting.
I20260128 15:03:04.265852 21788 tablet_bootstrap.cc:654] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:04.266227 21789 tablet_bootstrap.cc:654] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:04.266824 21788 tablet_bootstrap.cc:492] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e: No bootstrap required, opened a new log
I20260128 15:03:04.266885 21788 ts_tablet_manager.cc:1403] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260128 15:03:04.267103 21792 tablet_bootstrap.cc:492] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21: Bootstrap starting.
I20260128 15:03:04.267300 21789 tablet_bootstrap.cc:492] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe: No bootstrap required, opened a new log
I20260128 15:03:04.267385 21789 ts_tablet_manager.cc:1403] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260128 15:03:04.267331 21788 raft_consensus.cc:359] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } }
I20260128 15:03:04.267421 21788 raft_consensus.cc:385] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:04.267449 21788 raft_consensus.cc:740] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e25f3cc9ae44aba196ed92a3adfb0e, State: Initialized, Role: FOLLOWER
I20260128 15:03:04.267596 21788 consensus_queue.cc:260] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } }
I20260128 15:03:04.267781 21789 raft_consensus.cc:359] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } }
I20260128 15:03:04.267876 21789 raft_consensus.cc:385] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:04.267911 21789 raft_consensus.cc:740] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc94066dcfc943bdb867b2d77a4b8efe, State: Initialized, Role: FOLLOWER
I20260128 15:03:04.268023 21789 consensus_queue.cc:260] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } }
I20260128 15:03:04.268169 21792 tablet_bootstrap.cc:654] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:04.268210 21788 ts_tablet_manager.cc:1434] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:03:04.268528 21789 ts_tablet_manager.cc:1434] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:03:04.269527 21792 tablet_bootstrap.cc:492] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21: No bootstrap required, opened a new log
I20260128 15:03:04.269600 21792 ts_tablet_manager.cc:1403] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 15:03:04.270004 21792 raft_consensus.cc:359] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } }
I20260128 15:03:04.270098 21792 raft_consensus.cc:385] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:04.270130 21792 raft_consensus.cc:740] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc2239548eb481580722ea108a90e21, State: Initialized, Role: FOLLOWER
I20260128 15:03:04.270255 21792 consensus_queue.cc:260] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } }
I20260128 15:03:04.270570 21792 ts_tablet_manager.cc:1434] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:03:04.270798 21693 heartbeater.cc:499] Master 127.13.88.188:44165 was elected leader, sending a full tablet report...
I20260128 15:03:04.270877 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.269457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.131:36665)
0128 15:03:04.270763 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.270817 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1234}
I20260128 15:03:04.270951 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.270707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.130:57603)
0128 15:03:04.270839 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.270886 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260128 15:03:04.270951 21557 heartbeater.cc:499] Master 127.13.88.188:44165 was elected leader, sending a full tablet report...
I20260128 15:03:04.271711 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.271379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.130:48499)
0128 15:03:04.271603 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.271647 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260128 15:03:04.272070 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.271765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.129:46713)
0128 15:03:04.271964 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.272007 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260128 15:03:04.272933 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.272689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.131:33451)
0128 15:03:04.272827 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.272871 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260128 15:03:04.273648 21421 heartbeater.cc:499] Master 127.13.88.188:44165 was elected leader, sending a full tablet report...
W20260128 15:03:04.285069 21560 tablet.cc:2378] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 15:03:04.304464 21796 raft_consensus.cc:493] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:03:04.304590 21796 raft_consensus.cc:515] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } }
I20260128 15:03:04.305609 21796 leader_election.cc:290] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5e25f3cc9ae44aba196ed92a3adfb0e (127.13.88.130:41581), fc94066dcfc943bdb867b2d77a4b8efe (127.13.88.131:45859)
I20260128 15:03:04.308588 21797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.305823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:45859 (local address 127.13.88.129:42193)
0128 15:03:04.306187 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.306198 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:04.306292 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:04.306521 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:04.306524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:04.306540 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:04.306787 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:04.306793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.307491 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:04.307495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:04.308176 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:04.308185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.308284 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:04.308306 (+ 22us) client_negotiation.cc:769] Sending connection context
0128 15:03:04.308354 (+ 48us) client_negotiation.cc:240] Negotiation successful
0128 15:03:04.308411 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":53,"thread_start_us":134,"threads_started":1}
I20260128 15:03:04.308995 21787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.305928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:42193 (local address 127.13.88.131:45859)
0128 15:03:04.306051 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:04.306055 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:04.306277 (+ 222us) server_negotiation.cc:408] Connection header received
0128 15:03:04.306388 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:04.306390 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:04.306440 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:04.306503 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:04.306898 (+ 395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.307385 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:04.308286 (+ 901us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.308781 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:04.308819 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:04.308866 (+ 47us) server_negotiation.cc:300] Negotiation successful
0128 15:03:04.308910 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260128 15:03:04.309228 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.305816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:41581 (local address 127.13.88.129:36615)
0128 15:03:04.306032 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.306046 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:04.306160 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:04.306385 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:04.306395 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:04.306411 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:04.306874 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:04.306880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.307586 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:04.307588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:04.308259 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:04.308268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.308383 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:04.308441 (+ 58us) client_negotiation.cc:769] Sending connection context
0128 15:03:04.308493 (+ 52us) client_negotiation.cc:240] Negotiation successful
0128 15:03:04.308558 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260128 15:03:04.309295 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.305892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:36615 (local address 127.13.88.130:41581)
0128 15:03:04.306045 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:04.306049 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:04.306127 (+ 78us) server_negotiation.cc:408] Connection header received
0128 15:03:04.306262 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:04.306266 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:04.306307 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:04.306366 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:04.306994 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.307461 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:04.308643 (+ 1182us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.309040 (+ 397us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:04.309077 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:04.309127 (+ 50us) server_negotiation.cc:300] Negotiation successful
0128 15:03:04.309182 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260128 15:03:04.309381 21647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0eb31e7e746b493e839dd10943182c97" candidate_uuid: "3fc2239548eb481580722ea108a90e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" is_pre_election: true
I20260128 15:03:04.309527 21647 raft_consensus.cc:2468] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc2239548eb481580722ea108a90e21 in term 0.
I20260128 15:03:04.309747 21511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0eb31e7e746b493e839dd10943182c97" candidate_uuid: "3fc2239548eb481580722ea108a90e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" is_pre_election: true
I20260128 15:03:04.309860 21310 leader_election.cc:304] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc2239548eb481580722ea108a90e21, fc94066dcfc943bdb867b2d77a4b8efe; no voters:
I20260128 15:03:04.309950 21511 raft_consensus.cc:2468] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fc2239548eb481580722ea108a90e21 in term 0.
I20260128 15:03:04.310106 21796 raft_consensus.cc:2804] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:03:04.310174 21796 raft_consensus.cc:493] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:03:04.310258 21796 raft_consensus.cc:3060] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:04.311103 21796 raft_consensus.cc:515] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } }
I20260128 15:03:04.311430 21796 leader_election.cc:290] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [CANDIDATE]: Term 1 election: Requested vote from peers e5e25f3cc9ae44aba196ed92a3adfb0e (127.13.88.130:41581), fc94066dcfc943bdb867b2d77a4b8efe (127.13.88.131:45859)
I20260128 15:03:04.311712 21511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0eb31e7e746b493e839dd10943182c97" candidate_uuid: "3fc2239548eb481580722ea108a90e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e"
I20260128 15:03:04.311774 21647 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0eb31e7e746b493e839dd10943182c97" candidate_uuid: "3fc2239548eb481580722ea108a90e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc94066dcfc943bdb867b2d77a4b8efe"
I20260128 15:03:04.311815 21511 raft_consensus.cc:3060] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:04.311882 21647 raft_consensus.cc:3060] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:04.312661 21511 raft_consensus.cc:2468] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc2239548eb481580722ea108a90e21 in term 1.
I20260128 15:03:04.312669 21647 raft_consensus.cc:2468] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fc2239548eb481580722ea108a90e21 in term 1.
I20260128 15:03:04.312903 21310 leader_election.cc:304] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc2239548eb481580722ea108a90e21, fc94066dcfc943bdb867b2d77a4b8efe; no voters:
I20260128 15:03:04.313094 21796 raft_consensus.cc:2804] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:03:04.313289 21796 raft_consensus.cc:697] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [term 1 LEADER]: Becoming Leader. State: Replica: 3fc2239548eb481580722ea108a90e21, State: Running, Role: LEADER
I20260128 15:03:04.313526 21796 consensus_queue.cc:237] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } } peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } }
I20260128 15:03:04.314033 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.313804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.129:39135)
0128 15:03:04.313923 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.313970 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260128 15:03:04.315713 21222 catalog_manager.cc:5654] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fc2239548eb481580722ea108a90e21 (127.13.88.129). New cstate: current_term: 1 leader_uuid: "3fc2239548eb481580722ea108a90e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fc2239548eb481580722ea108a90e21" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 33663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 } health_report { overall_health: UNKNOWN } } }
I20260128 15:03:04.327101 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.324581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40280 (local address 127.13.88.129:33663)
0128 15:03:04.324685 (+ 104us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:04.324688 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:04.324700 (+ 12us) server_negotiation.cc:408] Connection header received
0128 15:03:04.324811 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:04.324814 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:04.324861 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:04.324933 (+ 72us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:03:04.325181 (+ 248us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.325638 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:04.326220 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:04.326345 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:04.326388 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:03:04.326813 (+ 425us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:03:04.326894 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:04.326972 (+ 78us) server_negotiation.cc:300] Negotiation successful
0128 15:03:04.327011 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260128 15:03:04.329471 21647 raft_consensus.cc:1275] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe [term 1 FOLLOWER]: Refusing update from remote peer 3fc2239548eb481580722ea108a90e21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:03:04.329741 21511 raft_consensus.cc:1275] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e [term 1 FOLLOWER]: Refusing update from remote peer 3fc2239548eb481580722ea108a90e21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:03:04.329934 21798 consensus_queue.cc:1048] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc94066dcfc943bdb867b2d77a4b8efe" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 45859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:03:04.330222 21796 consensus_queue.cc:1048] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e25f3cc9ae44aba196ed92a3adfb0e" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 41581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:03:04.331540 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.331308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.130:35977)
0128 15:03:04.331418 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.331469 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260128 15:03:04.332608 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.331979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.131:52457)
0128 15:03:04.332486 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.332541 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405}
I20260128 15:03:04.334604 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.334367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.129:59457)
0128 15:03:04.334485 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.334536 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260128 15:03:04.335309 21804 mvcc.cc:204] Tried to move back new op lower bound from 7248333145409589248 to 7248333145350807552. Current Snapshot: MvccSnapshot[applied={T|T < 7248333145409589248}]
I20260128 15:03:04.335409 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.335152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.130:48035)
0128 15:03:04.335296 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.335341 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260128 15:03:04.341030 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.340778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41873 (local address 127.13.88.131:45507)
0128 15:03:04.340903 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.340956 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:41873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260128 15:03:04.342339 21224 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:03:04.342482 21224 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:03:04.346050 21224 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c103d7f4e85045059b10b02c5788eaba: Sending DeleteTablet for 3 replicas of tablet 0eb31e7e746b493e839dd10943182c97
I20260128 15:03:04.346693 21355 tablet_service.cc:1552] Processing DeleteTablet for tablet 0eb31e7e746b493e839dd10943182c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:04 UTC) from {username='oryx', principal='oryx/127.13.88.188@KRBTEST.COM'} at 127.0.0.1:40272
I20260128 15:03:04.346719 21491 tablet_service.cc:1552] Processing DeleteTablet for tablet 0eb31e7e746b493e839dd10943182c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:04 UTC) from {username='oryx', principal='oryx/127.13.88.188@KRBTEST.COM'} at 127.0.0.1:49454
I20260128 15:03:04.346792 21627 tablet_service.cc:1552] Processing DeleteTablet for tablet 0eb31e7e746b493e839dd10943182c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:04 UTC) from {username='oryx', principal='oryx/127.13.88.188@KRBTEST.COM'} at 127.0.0.1:36650
I20260128 15:03:04.347288 21813 tablet_replica.cc:333] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe: stopping tablet replica
I20260128 15:03:04.347456 21813 raft_consensus.cc:2243] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:03:04.347591 21813 raft_consensus.cc:2272] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:04.347671 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 21292
I20260128 15:03:04.347805 21811 tablet_replica.cc:333] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21: stopping tablet replica
I20260128 15:03:04.347890 21811 raft_consensus.cc:2243] T 0eb31e7e746b493e839dd10943182c97 P 3fc2239548eb481580722ea108a90e21 [term 1 LEADER]: Raft consensus shutting down.
I20260128 15:03:04.348292 21812 tablet_replica.cc:333] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e: stopping tablet replica
I20260128 15:03:04.348392 21812 raft_consensus.cc:2243] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:03:04.348532 21812 raft_consensus.cc:2272] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:04.349148 21813 ts_tablet_manager.cc:1916] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:04.350315 21812 ts_tablet_manager.cc:1916] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:04.351204 21813 ts_tablet_manager.cc:1929] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:03:04.351303 21813 log.cc:1199] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-2/wal/wals/0eb31e7e746b493e839dd10943182c97
I20260128 15:03:04.351562 21813 ts_tablet_manager.cc:1950] T 0eb31e7e746b493e839dd10943182c97 P fc94066dcfc943bdb867b2d77a4b8efe: Deleting consensus metadata
I20260128 15:03:04.352252 21210 catalog_manager.cc:4985] TS fc94066dcfc943bdb867b2d77a4b8efe (127.13.88.131:45859): tablet 0eb31e7e746b493e839dd10943182c97 (table test-table [id=574387a8993f41cb89ccc037c673ea7a]) successfully deleted
I20260128 15:03:04.352581 21812 ts_tablet_manager.cc:1929] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:03:04.352643 21812 log.cc:1199] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1769612519049359-13666-0/minicluster-data/ts-1/wal/wals/0eb31e7e746b493e839dd10943182c97
I20260128 15:03:04.352860 21812 ts_tablet_manager.cc:1950] T 0eb31e7e746b493e839dd10943182c97 P e5e25f3cc9ae44aba196ed92a3adfb0e: Deleting consensus metadata
I20260128 15:03:04.353447 21209 catalog_manager.cc:4985] TS e5e25f3cc9ae44aba196ed92a3adfb0e (127.13.88.130:41581): tablet 0eb31e7e746b493e839dd10943182c97 (table test-table [id=574387a8993f41cb89ccc037c673ea7a]) successfully deleted
I20260128 15:03:04.354605 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 21428
I20260128 15:03:04.354784 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:04.354546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:33663 (local address 127.0.0.1:40290)
0128 15:03:04.354663 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:04.354713 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.129:33663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260128 15:03:04.355190 21209 catalog_manager.cc:4712] TS 3fc2239548eb481580722ea108a90e21 (127.13.88.129:33663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0eb31e7e746b493e839dd10943182c97: Network error: Client connection negotiation failed: client connection to 127.13.88.129:33663: connect: Connection refused (error 111)
I20260128 15:03:04.360709 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 21564
I20260128 15:03:04.365574 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 20901
I20260128 15:03:04.370322 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 21012
2026-01-28T15:03:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8639 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:03:04 dist-test-slave-m28t krb5kdc[21816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:03:04 dist-test-slave-m28t krb5kdc[21816](info): set up 2 sockets
Jan 28 15:03:04 dist-test-slave-m28t krb5kdc[21816](info): commencing operation
krb5kdc: starting...
W20260128 15:03:06.193020 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.801s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Jan 28 15:03:06 dist-test-slave-m28t krb5kdc[21816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:03:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:03:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab.
I20260128 15:03:06.318895 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:41065
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:39717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:41065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:03:06.424134 21832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:06.424382 21832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:06.424484 21832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:06.427759 21832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:03:06.427832 21832 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260128 15:03:06.427857 21832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:06.427884 21832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:06.427918 21832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:03:06.427950 21832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:03:06.432332 21832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:39717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:41065
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:41065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:06.433465 21832 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:06.434326 21832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:06.440523 21838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:06.440513 21837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:06.440512 21832 server_base.cc:1060] running on GCE node
W20260128 15:03:06.440526 21840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:06.441223 21832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:06.442206 21832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:06.443368 21832 hybrid_clock.cc:648] HybridClock initialized: now 1769612586443340 us; error 43 us; skew 500 ppm
Jan 28 15:03:06 dist-test-slave-m28t krb5kdc[21816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612586, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:06.445902 21832 init.cc:377] Logged in from keytab as oryx/127.13.88.190@KRBTEST.COM (short username oryx)
I20260128 15:03:06.446954 21832 webserver.cc:492] Webserver started at http://127.13.88.190:39569/ using document root <none> and password file <none>
I20260128 15:03:06.447497 21832 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:06.447549 21832 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:06.447727 21832 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:06.449508 21832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "24a668aa02bd496e84c8cace1b6f5bf2"
format_stamp: "Formatted at 2026-01-28 15:03:06 on dist-test-slave-m28t"
server_key: "0caed5fa5bd0d93a4c14e952c137f9ad"
server_key_iv: "6121a9c4657613f4d1ee919532dee7e0"
server_key_version: "encryptionkey@0"
I20260128 15:03:06.449966 21832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "24a668aa02bd496e84c8cace1b6f5bf2"
format_stamp: "Formatted at 2026-01-28 15:03:06 on dist-test-slave-m28t"
server_key: "0caed5fa5bd0d93a4c14e952c137f9ad"
server_key_iv: "6121a9c4657613f4d1ee919532dee7e0"
server_key_version: "encryptionkey@0"
I20260128 15:03:06.453276 21832 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260128 15:03:06.455380 21847 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:06.456372 21832 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:03:06.456503 21832 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "24a668aa02bd496e84c8cace1b6f5bf2"
format_stamp: "Formatted at 2026-01-28 15:03:06 on dist-test-slave-m28t"
server_key: "0caed5fa5bd0d93a4c14e952c137f9ad"
server_key_iv: "6121a9c4657613f4d1ee919532dee7e0"
server_key_version: "encryptionkey@0"
I20260128 15:03:06.456589 21832 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:06.480813 21832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:06.483983 21832 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:06.484169 21832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:06.491839 21832 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:41065
I20260128 15:03:06.492031 21909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:41065 every 8 connection(s)
I20260128 15:03:06.492893 21832 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:03:06.494071 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 21832
I20260128 15:03:06.494180 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:03:06.494410 13666 external_mini_cluster.cc:1468] Setting key 2684ffd071faf310663ec378eb1dd387
I20260128 15:03:06.495937 21910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:06.500869 21910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2: Bootstrap starting.
Jan 28 15:03:06 dist-test-slave-m28t krb5kdc[21816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM
I20260128 15:03:06.502892 21910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:06.503587 21910 log.cc:826] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2: Log is configured to *not* fsync() on all Append() calls
I20260128 15:03:06.505211 21910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2: No bootstrap required, opened a new log
I20260128 15:03:06.506330 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:06.495693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.13.88.190:41065)
0128 15:03:06.496239 (+ 546us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:06.496251 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:06.496301 (+ 50us) server_negotiation.cc:408] Connection header received
0128 15:03:06.497043 (+ 742us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:06.497061 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:06.497344 (+ 283us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:06.497645 (+ 301us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:06.498499 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:06.499256 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:06.499878 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:06.500131 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:06.502125 (+ 1994us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:06.502164 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:06.502174 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:06.502205 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:06.503855 (+ 1650us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:06.504238 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:06.504243 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:06.504247 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:06.504319 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:06.504574 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:06.504577 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:06.504580 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:06.504900 (+ 320us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:06.505093 (+ 193us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:06.505371 (+ 278us) server_negotiation.cc:300] Negotiation successful
0128 15:03:06.505611 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":128,"threads_started":1}
I20260128 15:03:06.507601 21910 raft_consensus.cc:359] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a668aa02bd496e84c8cace1b6f5bf2" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41065 } }
I20260128 15:03:06.507774 21910 raft_consensus.cc:385] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:06.507839 21910 raft_consensus.cc:740] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a668aa02bd496e84c8cace1b6f5bf2, State: Initialized, Role: FOLLOWER
I20260128 15:03:06.508230 21910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a668aa02bd496e84c8cace1b6f5bf2" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41065 } }
I20260128 15:03:06.508351 21910 raft_consensus.cc:399] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:03:06.508407 21910 raft_consensus.cc:493] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:03:06.508518 21910 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:06.509233 21910 raft_consensus.cc:515] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a668aa02bd496e84c8cace1b6f5bf2" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41065 } }
I20260128 15:03:06.509537 21910 leader_election.cc:304] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24a668aa02bd496e84c8cace1b6f5bf2; no voters:
I20260128 15:03:06.509794 21910 leader_election.cc:290] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:03:06.509878 21915 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:03:06.510057 21915 raft_consensus.cc:697] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [term 1 LEADER]: Becoming Leader. State: Replica: 24a668aa02bd496e84c8cace1b6f5bf2, State: Running, Role: LEADER
I20260128 15:03:06.510372 21915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a668aa02bd496e84c8cace1b6f5bf2" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41065 } }
I20260128 15:03:06.510744 21910 sys_catalog.cc:565] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:03:06.511742 21917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24a668aa02bd496e84c8cace1b6f5bf2. Latest consensus state: current_term: 1 leader_uuid: "24a668aa02bd496e84c8cace1b6f5bf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a668aa02bd496e84c8cace1b6f5bf2" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41065 } } }
I20260128 15:03:06.511854 21917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:06.511979 21916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24a668aa02bd496e84c8cace1b6f5bf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a668aa02bd496e84c8cace1b6f5bf2" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 41065 } } }
I20260128 15:03:06.512120 21916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2 [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:06.513095 21924 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:03:06.515427 21924 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:03:06.520098 21924 catalog_manager.cc:1357] Generated new cluster ID: 883a2b9b9050479ea7ebcec5f71a5354
I20260128 15:03:06.520150 21924 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:03:06.545784 21924 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:03:06.546738 21924 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:03:06.555644 21924 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2: Generated new TSK 0
I20260128 15:03:06.556337 21924 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab.
I20260128 15:03:06.604595 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:41065
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:39717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:03:06.714314 21938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:06.714514 21938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:06.714572 21938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:06.717741 21938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:06.717798 21938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:06.717849 21938 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 15:03:06.717890 21938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:03:06.722005 21938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:39717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.88.190:41065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:06.722962 21938 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:06.723738 21938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:06.729894 21946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:06.729899 21943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:06.729993 21938 server_base.cc:1060] running on GCE node
W20260128 15:03:06.729907 21944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:06.730490 21938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:06.730984 21938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:06.732128 21938 hybrid_clock.cc:648] HybridClock initialized: now 1769612586732112 us; error 31 us; skew 500 ppm
Jan 28 15:03:06 dist-test-slave-m28t krb5kdc[21816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612586, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:06.734452 21938 init.cc:377] Logged in from keytab as oryx/127.13.88.129@KRBTEST.COM (short username oryx)
I20260128 15:03:06.735388 21938 webserver.cc:492] Webserver started at http://127.13.88.129:38361/ using document root <none> and password file <none>
I20260128 15:03:06.735877 21938 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:06.735924 21938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:06.736083 21938 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:06.737735 21938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "d88dca1fa2c84717a0e65c9e2d430d19"
format_stamp: "Formatted at 2026-01-28 15:03:06 on dist-test-slave-m28t"
server_key: "98daa202ff13df0983367615bf2d9951"
server_key_iv: "6407a385e39a631b5e8ace2e3ce35279"
server_key_version: "encryptionkey@0"
I20260128 15:03:06.738168 21938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "d88dca1fa2c84717a0e65c9e2d430d19"
format_stamp: "Formatted at 2026-01-28 15:03:06 on dist-test-slave-m28t"
server_key: "98daa202ff13df0983367615bf2d9951"
server_key_iv: "6407a385e39a631b5e8ace2e3ce35279"
server_key_version: "encryptionkey@0"
I20260128 15:03:06.741299 21938 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260128 15:03:06.743196 21953 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:06.744071 21938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:03:06.744161 21938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "d88dca1fa2c84717a0e65c9e2d430d19"
format_stamp: "Formatted at 2026-01-28 15:03:06 on dist-test-slave-m28t"
server_key: "98daa202ff13df0983367615bf2d9951"
server_key_iv: "6407a385e39a631b5e8ace2e3ce35279"
server_key_version: "encryptionkey@0"
I20260128 15:03:06.744249 21938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:06.757292 21938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:06.759478 21938 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:06.759629 21938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:06.761067 21938 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:06.761118 21938 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:06.761166 21938 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:06.761193 21938 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 28 15:03:06 dist-test-slave-m28t krb5kdc[21816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612586, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM
I20260128 15:03:06.771664 21938 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:42207
I20260128 15:03:06.771718 22068 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:42207 every 8 connection(s)
I20260128 15:03:06.772301 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:06.761508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:35179 (local address 127.13.88.190:41065)
0128 15:03:06.761647 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:06.761652 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:06.762337 (+ 685us) server_negotiation.cc:408] Connection header received
0128 15:03:06.763371 (+ 1034us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:06.763374 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:06.763422 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:06.763517 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:06.765163 (+ 1646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:06.765767 (+ 604us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:06.766496 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:06.766688 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:06.768727 (+ 2039us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:06.768747 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:06.768749 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:06.768775 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:06.770252 (+ 1477us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:06.770848 (+ 596us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:06.770852 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:06.770853 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:06.770899 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:06.771170 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:06.771173 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:06.771174 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:06.771326 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:06.771413 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:06.772003 (+ 590us) server_negotiation.cc:300] Negotiation successful
0128 15:03:06.772136 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260128 15:03:06.772627 21938 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:03:06.772912 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:06.761748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41065 (local address 127.13.88.129:35179)
0128 15:03:06.762184 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:06.762229 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:06.763172 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:06.763696 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:06.763707 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:06.764222 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:06.765017 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:06.765033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:06.765885 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:06.765889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:06.766374 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:06.766384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:06.766590 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:06.767161 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:06.767196 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:06.768584 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:06.770383 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:06.770389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:06.770401 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:06.770745 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:06.770993 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:06.770997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:06.770999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:06.771088 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:06.771479 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:06.771485 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:06.771744 (+ 259us) client_negotiation.cc:769] Sending connection context
0128 15:03:06.771952 (+ 208us) client_negotiation.cc:240] Negotiation successful
0128 15:03:06.772199 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":94,"threads_started":1}
I20260128 15:03:06.774165 22069 heartbeater.cc:344] Connected to a master server at 127.13.88.190:41065
I20260128 15:03:06.774299 22069 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:06.774735 22069 heartbeater.cc:507] Master 127.13.88.190:41065 requested a full tablet report, sending...
I20260128 15:03:06.776154 21863 ts_manager.cc:194] Registered new tserver with Master: d88dca1fa2c84717a0e65c9e2d430d19 (127.13.88.129:42207)
I20260128 15:03:06.777607 21863 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.88.129@KRBTEST.COM'} at 127.13.88.129:35179
I20260128 15:03:06.779901 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 21938
I20260128 15:03:06.780007 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:03:06.780241 13666 external_mini_cluster.cc:1468] Setting key b2f08828d539f523a91c5c3f9507b37b
I20260128 15:03:06.781767 22075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:06.775894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41065 (local address 127.13.88.129:41993)
0128 15:03:06.776236 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:06.776250 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:06.776372 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:06.776620 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:06.776622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:06.776792 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:06.777061 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:06.777069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:06.777795 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:06.777799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:06.778268 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:06.778276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:06.778376 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:06.778829 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:06.778845 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:06.779109 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:06.780613 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:06.780616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:06.780619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:06.780775 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:06.780980 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:06.780983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:06.780985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:06.781037 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:06.781361 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:06.781364 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:06.781450 (+ 86us) client_negotiation.cc:769] Sending connection context
0128 15:03:06.781552 (+ 102us) client_negotiation.cc:240] Negotiation successful
0128 15:03:06.781646 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":101,"threads_started":1}
I20260128 15:03:06.781874 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:06.775953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:41993 (local address 127.13.88.190:41065)
0128 15:03:06.776049 (+ 96us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:06.776052 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:06.776311 (+ 259us) server_negotiation.cc:408] Connection header received
0128 15:03:06.776487 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:06.776489 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:06.776529 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:06.776585 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:06.777201 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:06.777680 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:06.778374 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:06.778819 (+ 445us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:06.779215 (+ 396us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:06.779230 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:06.779233 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:06.779256 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:06.780524 (+ 1268us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:06.780864 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:06.780866 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:06.780868 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:06.780909 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:06.781105 (+ 196us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:06.781107 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:06.781108 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:06.781269 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:06.781355 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:06.781627 (+ 272us) server_negotiation.cc:300] Negotiation successful
0128 15:03:06.781740 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
WARNING: no policy specified for oryx/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab.
I20260128 15:03:06.820849 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:41065
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:39717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:03:06.927003 22080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:06.927227 22080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:06.927309 22080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:06.930601 22080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:06.930661 22080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:06.930730 22080 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 15:03:06.930790 22080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:03:06.934898 22080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:39717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.88.190:41065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:06.935902 22080 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:06.936771 22080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:06.942974 22088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:06.943053 22085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:06.942991 22086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:06.943256 22080 server_base.cc:1060] running on GCE node
I20260128 15:03:06.943634 22080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:06.944154 22080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:06.945303 22080 hybrid_clock.cc:648] HybridClock initialized: now 1769612586945288 us; error 32 us; skew 500 ppm
Jan 28 15:03:06 dist-test-slave-m28t krb5kdc[21816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612586, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:06.947649 22080 init.cc:377] Logged in from keytab as oryx/127.13.88.130@KRBTEST.COM (short username oryx)
I20260128 15:03:06.948689 22080 webserver.cc:492] Webserver started at http://127.13.88.130:34837/ using document root <none> and password file <none>
I20260128 15:03:06.949208 22080 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:06.949256 22080 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:06.949426 22080 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:06.951076 22080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "0d476c4efff04de7aa1c66ae0992c07f"
format_stamp: "Formatted at 2026-01-28 15:03:06 on dist-test-slave-m28t"
server_key: "e30d752c47f2cfff616bfc95a635e494"
server_key_iv: "50b163b64c80cba0460b260498fb00ab"
server_key_version: "encryptionkey@0"
I20260128 15:03:06.951524 22080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "0d476c4efff04de7aa1c66ae0992c07f"
format_stamp: "Formatted at 2026-01-28 15:03:06 on dist-test-slave-m28t"
server_key: "e30d752c47f2cfff616bfc95a635e494"
server_key_iv: "50b163b64c80cba0460b260498fb00ab"
server_key_version: "encryptionkey@0"
I20260128 15:03:06.954742 22080 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260128 15:03:06.956920 22095 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:06.957940 22080 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260128 15:03:06.958035 22080 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "0d476c4efff04de7aa1c66ae0992c07f"
format_stamp: "Formatted at 2026-01-28 15:03:06 on dist-test-slave-m28t"
server_key: "e30d752c47f2cfff616bfc95a635e494"
server_key_iv: "50b163b64c80cba0460b260498fb00ab"
server_key_version: "encryptionkey@0"
I20260128 15:03:06.958112 22080 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:07.002049 22080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:07.004683 22080 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:07.004830 22080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:07.006330 22080 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:07.006379 22080 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:07.006430 22080 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:07.006461 22080 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jan 28 15:03:07 dist-test-slave-m28t krb5kdc[21816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612586, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM
I20260128 15:03:07.017150 22080 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:45523
I20260128 15:03:07.017174 22210 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:45523 every 8 connection(s)
I20260128 15:03:07.017346 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.006880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:59193 (local address 127.13.88.190:41065)
0128 15:03:07.006990 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:07.006993 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:07.007847 (+ 854us) server_negotiation.cc:408] Connection header received
0128 15:03:07.008837 (+ 990us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:07.008841 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:07.008895 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:07.008965 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:07.010318 (+ 1353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.010851 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.011599 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.011812 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.014082 (+ 2270us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:07.014102 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:07.014106 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:07.014140 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:07.015544 (+ 1404us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:07.015965 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:07.015968 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:07.015969 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:07.016012 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:07.016278 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:07.016281 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:07.016283 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:07.016483 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:07.016568 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:07.017061 (+ 493us) server_negotiation.cc:300] Negotiation successful
0128 15:03:07.017187 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260128 15:03:07.017971 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.007231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41065 (local address 127.13.88.130:59193)
0128 15:03:07.007719 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:07.007756 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:07.008560 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:07.009136 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:07.009147 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:07.009638 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:07.010168 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.010181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.010961 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.010964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:07.011425 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.011445 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.011645 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.012232 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:07.012267 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:07.013940 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:07.015654 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:07.015660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:07.015669 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:07.015874 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:07.016088 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:07.016091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:07.016104 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:07.016195 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:07.016577 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:07.016583 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:07.016799 (+ 216us) client_negotiation.cc:769] Sending connection context
0128 15:03:07.017009 (+ 210us) client_negotiation.cc:240] Negotiation successful
0128 15:03:07.017277 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":131,"threads_started":1}
I20260128 15:03:07.018406 22080 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
I20260128 15:03:07.019645 22211 heartbeater.cc:344] Connected to a master server at 127.13.88.190:41065
I20260128 15:03:07.019784 22211 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:07.020318 22211 heartbeater.cc:507] Master 127.13.88.190:41065 requested a full tablet report, sending...
I20260128 15:03:07.020653 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 22080
I20260128 15:03:07.020751 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:03:07.021008 13666 external_mini_cluster.cc:1468] Setting key c9275f066dd8e5d54b41d6bf8c1fcebe
I20260128 15:03:07.021297 21863 ts_manager.cc:194] Registered new tserver with Master: 0d476c4efff04de7aa1c66ae0992c07f (127.13.88.130:45523)
I20260128 15:03:07.021886 21863 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.88.130@KRBTEST.COM'} at 127.13.88.130:59193
I20260128 15:03:07.027801 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.022146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:55899 (local address 127.13.88.190:41065)
0128 15:03:07.022263 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:07.022267 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:07.022280 (+ 13us) server_negotiation.cc:408] Connection header received
0128 15:03:07.022310 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:07.022313 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:07.022349 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:07.022437 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:07.022929 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.023620 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.024268 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.024486 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.025169 (+ 683us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:07.025185 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:07.025187 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:07.025208 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:07.026462 (+ 1254us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:07.026816 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:07.026819 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:07.026820 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:07.026861 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:07.027058 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:07.027061 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:07.027064 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:07.027232 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:07.027319 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:07.027546 (+ 227us) server_negotiation.cc:300] Negotiation successful
0128 15:03:07.027649 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260128 15:03:07.027828 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.021343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41065 (local address 127.13.88.130:55899)
0128 15:03:07.021584 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:07.021598 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:07.021686 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:07.022446 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:07.022448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:07.022592 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:07.022817 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.022826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.023752 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.023756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:07.024152 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.024159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.024263 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.024777 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:07.024789 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:07.025051 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:07.026569 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:07.026572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:07.026575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:07.026730 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:07.026938 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:07.026941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:07.026942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:07.026985 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:07.027320 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:07.027324 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:07.027399 (+ 75us) client_negotiation.cc:769] Sending connection context
0128 15:03:07.027492 (+ 93us) client_negotiation.cc:240] Negotiation successful
0128 15:03:07.027626 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab.
I20260128 15:03:07.068876 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:41065
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:39717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:03:07.174432 22222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:07.174647 22222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:07.174705 22222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:07.177978 22222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:07.178049 22222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:07.178114 22222 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260128 15:03:07.178177 22222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:03:07.182391 22222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:39717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/krb5kdc/oryx.keytab
--principal=oryx/127.13.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.88.190:41065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:07.183521 22222 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:07.184409 22222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:07.191326 22227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:07.191394 22228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:07.191391 22230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:07.191591 22222 server_base.cc:1060] running on GCE node
I20260128 15:03:07.191962 22222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:07.192553 22222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:07.193712 22222 hybrid_clock.cc:648] HybridClock initialized: now 1769612587193693 us; error 37 us; skew 500 ppm
Jan 28 15:03:07 dist-test-slave-m28t krb5kdc[21816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612587, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:07.196292 22222 init.cc:377] Logged in from keytab as oryx/127.13.88.131@KRBTEST.COM (short username oryx)
I20260128 15:03:07.197346 22222 webserver.cc:492] Webserver started at http://127.13.88.131:44859/ using document root <none> and password file <none>
I20260128 15:03:07.197872 22222 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:07.197921 22222 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:07.198096 22222 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:07.199761 22222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d"
format_stamp: "Formatted at 2026-01-28 15:03:07 on dist-test-slave-m28t"
server_key: "f68262772ec228665702aa6b023e4bf5"
server_key_iv: "a4dd69bf48e8a9da2157718eb8085e6d"
server_key_version: "encryptionkey@0"
I20260128 15:03:07.200217 22222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d"
format_stamp: "Formatted at 2026-01-28 15:03:07 on dist-test-slave-m28t"
server_key: "f68262772ec228665702aa6b023e4bf5"
server_key_iv: "a4dd69bf48e8a9da2157718eb8085e6d"
server_key_version: "encryptionkey@0"
I20260128 15:03:07.203457 22222 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260128 15:03:07.205639 22237 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:07.206601 22222 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:03:07.206701 22222 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d"
format_stamp: "Formatted at 2026-01-28 15:03:07 on dist-test-slave-m28t"
server_key: "f68262772ec228665702aa6b023e4bf5"
server_key_iv: "a4dd69bf48e8a9da2157718eb8085e6d"
server_key_version: "encryptionkey@0"
I20260128 15:03:07.206784 22222 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:07.230566 22222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:07.233068 22222 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:07.233258 22222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:07.234754 22222 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:07.234810 22222 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:07.234863 22222 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:07.234903 22222 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jan 28 15:03:07 dist-test-slave-m28t krb5kdc[21816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612587, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM
I20260128 15:03:07.245405 22222 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:36433
I20260128 15:03:07.245437 22352 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:36433 every 8 connection(s)
I20260128 15:03:07.246457 22222 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
I20260128 15:03:07.246845 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.235297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:53033 (local address 127.13.88.190:41065)
0128 15:03:07.235441 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:07.235446 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:07.236226 (+ 780us) server_negotiation.cc:408] Connection header received
0128 15:03:07.237139 (+ 913us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:07.237143 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:07.237206 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:07.237298 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:07.238749 (+ 1451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.239522 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.240401 (+ 879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.240631 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.242990 (+ 2359us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:07.243008 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:07.243010 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:07.243036 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:07.245024 (+ 1988us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:07.245525 (+ 501us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:07.245529 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:07.245531 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:07.245584 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:07.245845 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:07.245849 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:07.245851 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:07.246025 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:07.246120 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:07.246565 (+ 445us) server_negotiation.cc:300] Negotiation successful
0128 15:03:07.246681 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260128 15:03:07.247546 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.235616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41065 (local address 127.13.88.131:53033)
0128 15:03:07.236057 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:07.236112 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:07.236935 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:07.237459 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:07.237477 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:07.237940 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:07.238607 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.238622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.239671 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.239676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:07.240269 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.240280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.240528 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.241097 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:07.241121 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:07.242834 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:07.245139 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:07.245144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:07.245155 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:07.245407 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:07.245668 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:07.245670 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:07.245671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:07.245776 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:07.246109 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:07.246114 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:07.246330 (+ 216us) client_negotiation.cc:769] Sending connection context
0128 15:03:07.246535 (+ 205us) client_negotiation.cc:240] Negotiation successful
0128 15:03:07.246794 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":103,"threads_started":1}
I20260128 15:03:07.248747 22353 heartbeater.cc:344] Connected to a master server at 127.13.88.190:41065
I20260128 15:03:07.248891 22353 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:07.249363 22353 heartbeater.cc:507] Master 127.13.88.190:41065 requested a full tablet report, sending...
I20260128 15:03:07.250309 21864 ts_manager.cc:194] Registered new tserver with Master: 4b4ced995cca48a8a1c2a1d4d6ae6f8d (127.13.88.131:36433)
I20260128 15:03:07.251005 21864 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.88.131@KRBTEST.COM'} at 127.13.88.131:53033
I20260128 15:03:07.255033 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 22222
I20260128 15:03:07.255151 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:03:07.255436 13666 external_mini_cluster.cc:1468] Setting key dca8485d04e8024c7d288041281461df
I20260128 15:03:07.255750 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.250591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:52467 (local address 127.13.88.190:41065)
0128 15:03:07.250715 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:07.250718 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:07.250840 (+ 122us) server_negotiation.cc:408] Connection header received
0128 15:03:07.250956 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:07.250958 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:07.250998 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:07.251063 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:07.251594 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.252050 (+ 456us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.252651 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.252823 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.253471 (+ 648us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:07.253487 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:07.253490 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:07.253511 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:07.254467 (+ 956us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:07.254795 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:07.254799 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:07.254801 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:07.254840 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:07.255025 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:07.255029 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:07.255031 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:07.255164 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:07.255263 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:07.255550 (+ 287us) server_negotiation.cc:300] Negotiation successful
0128 15:03:07.255645 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260128 15:03:07.255774 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.250521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41065 (local address 127.13.88.131:52467)
0128 15:03:07.250776 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:07.250790 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:07.250877 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:07.251093 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:07.251096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:07.251277 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:07.251485 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.251491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.252146 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.252150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:07.252555 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.252562 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.252658 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.253097 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:07.253109 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:07.253363 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:07.254561 (+ 1198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:07.254565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:07.254567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:07.254715 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:07.254911 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:07.254914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:07.254916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:07.254955 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:07.255275 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:07.255279 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:07.255371 (+ 92us) client_negotiation.cc:769] Sending connection context
0128 15:03:07.255483 (+ 112us) client_negotiation.cc:240] Negotiation successful
0128 15:03:07.255616 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260128 15:03:07.257822 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Jan 28 15:03:07 dist-test-slave-m28t krb5kdc[21816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jan 28 15:03:07 dist-test-slave-m28t krb5kdc[21816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:07 dist-test-slave-m28t krb5kdc[21816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
W20260128 15:03:07.277956 21913 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:07.272782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.13.88.190:41065)
0128 15:03:07.272917 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:07.272921 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:07.272932 (+ 11us) server_negotiation.cc:408] Connection header received
0128 15:03:07.272971 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:07.272974 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:07.273013 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:07.273080 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:07.273893 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.274379 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.275060 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.275247 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.277841 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43314: BlockingRecv error: recv got EOF from 127.0.0.1:43314 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jan 28 15:03:07 dist-test-slave-m28t krb5kdc[21816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM
I20260128 15:03:07.285424 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.279159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.13.88.190:41065)
0128 15:03:07.279268 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:07.279271 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:07.279358 (+ 87us) server_negotiation.cc:408] Connection header received
0128 15:03:07.279442 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:07.279444 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:07.279483 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:07.279549 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:07.280139 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.280658 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.281175 (+ 517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.281301 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.283060 (+ 1759us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:07.283079 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:07.283083 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:07.283111 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:07.284365 (+ 1254us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:07.284699 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:07.284702 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:07.284703 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:07.284743 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:07.284878 (+ 135us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:07.284879 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:07.284881 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:07.285003 (+ 122us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:07.285093 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:07.285228 (+ 135us) server_negotiation.cc:300] Negotiation successful
0128 15:03:07.285311 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260128 15:03:07.287740 21864 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 15:03:07.290145 21864 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:03:07.302181 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.298526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:45523 (local address 127.0.0.1:50542)
0128 15:03:07.298921 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:07.298933 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:07.299002 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:07.299702 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:07.299705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:07.299721 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:07.299921 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.299928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.300910 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.300914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:07.301719 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.301728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.301829 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.301884 (+ 55us) client_negotiation.cc:769] Sending connection context
0128 15:03:07.301965 (+ 81us) client_negotiation.cc:240] Negotiation successful
0128 15:03:07.302045 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":87,"threads_started":1}
I20260128 15:03:07.302500 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.298112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:42207 (local address 127.0.0.1:53318)
0128 15:03:07.298843 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:07.298855 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:07.298940 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:07.299711 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:07.299714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:07.299749 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:07.299947 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.299954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.301320 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.301324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:07.302005 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.302014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.302104 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.302121 (+ 17us) client_negotiation.cc:769] Sending connection context
0128 15:03:07.302165 (+ 44us) client_negotiation.cc:240] Negotiation successful
0128 15:03:07.302224 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"spinlock_wait_cycles":2432,"thread_start_us":85,"threads_started":1}
I20260128 15:03:07.302631 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.297905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:36433 (local address 127.0.0.1:55210)
0128 15:03:07.298605 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:07.298645 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:07.298773 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:07.299507 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:07.299512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:07.299533 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:07.299822 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.299830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.301277 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.301280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:07.302074 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.302081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.302168 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.302193 (+ 25us) client_negotiation.cc:769] Sending connection context
0128 15:03:07.302237 (+ 44us) client_negotiation.cc:240] Negotiation successful
0128 15:03:07.302519 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":62,"threads_started":1}
I20260128 15:03:07.303268 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.298926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.13.88.130:45523)
0128 15:03:07.299275 (+ 349us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:07.299280 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:07.299297 (+ 17us) server_negotiation.cc:408] Connection header received
0128 15:03:07.299362 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:07.299366 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:07.299507 (+ 141us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:07.299614 (+ 107us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:07.300026 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.300797 (+ 771us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.302379 (+ 1582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.302887 (+ 508us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.303003 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:07.303062 (+ 59us) server_negotiation.cc:300] Negotiation successful
0128 15:03:07.303141 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":94,"threads_started":1}
I20260128 15:03:07.303653 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.298019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.13.88.131:36433)
0128 15:03:07.298768 (+ 749us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:07.298775 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:07.298799 (+ 24us) server_negotiation.cc:408] Connection header received
0128 15:03:07.298956 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:07.298962 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:07.299151 (+ 189us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:07.299310 (+ 159us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:07.299948 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.301164 (+ 1216us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.302794 (+ 1630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.303276 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.303372 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:07.303434 (+ 62us) server_negotiation.cc:300] Negotiation successful
0128 15:03:07.303522 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":143,"threads_started":1}
I20260128 15:03:07.304054 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.298553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.13.88.129:42207)
0128 15:03:07.299157 (+ 604us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:07.299164 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:07.299185 (+ 21us) server_negotiation.cc:408] Connection header received
0128 15:03:07.299278 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:07.299284 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:07.299436 (+ 152us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:07.299564 (+ 128us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:07.300078 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.301232 (+ 1154us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.302742 (+ 1510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.303107 (+ 365us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.303189 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:07.303241 (+ 52us) server_negotiation.cc:300] Negotiation successful
0128 15:03:07.303305 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":68,"threads_started":1}
I20260128 15:03:07.306049 22144 tablet_service.cc:1505] Processing CreateTablet for tablet 0954cd59361144118c1a2cf0063bdca5 (DEFAULT_TABLE table=test-table [id=3f9df442f7054632bed2769a009810f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:07.306066 22287 tablet_service.cc:1505] Processing CreateTablet for tablet 0954cd59361144118c1a2cf0063bdca5 (DEFAULT_TABLE table=test-table [id=3f9df442f7054632bed2769a009810f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:07.306216 22003 tablet_service.cc:1505] Processing CreateTablet for tablet 0954cd59361144118c1a2cf0063bdca5 (DEFAULT_TABLE table=test-table [id=3f9df442f7054632bed2769a009810f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:07.306875 22144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0954cd59361144118c1a2cf0063bdca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:07.306907 22003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0954cd59361144118c1a2cf0063bdca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:07.306907 22287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0954cd59361144118c1a2cf0063bdca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:07.312211 22381 tablet_bootstrap.cc:492] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f: Bootstrap starting.
I20260128 15:03:07.313040 22383 tablet_bootstrap.cc:492] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Bootstrap starting.
I20260128 15:03:07.314832 22383 tablet_bootstrap.cc:654] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:07.314833 22381 tablet_bootstrap.cc:654] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:07.315783 22383 log.cc:826] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Log is configured to *not* fsync() on all Append() calls
I20260128 15:03:07.315783 22381 log.cc:826] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f: Log is configured to *not* fsync() on all Append() calls
I20260128 15:03:07.316329 22382 tablet_bootstrap.cc:492] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19: Bootstrap starting.
I20260128 15:03:07.317368 22381 tablet_bootstrap.cc:492] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f: No bootstrap required, opened a new log
I20260128 15:03:07.317495 22383 tablet_bootstrap.cc:492] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: No bootstrap required, opened a new log
I20260128 15:03:07.317549 22381 ts_tablet_manager.cc:1403] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260128 15:03:07.317665 22383 ts_tablet_manager.cc:1403] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260128 15:03:07.318246 22382 tablet_bootstrap.cc:654] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:07.318982 22382 log.cc:826] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19: Log is configured to *not* fsync() on all Append() calls
I20260128 15:03:07.320287 22381 raft_consensus.cc:359] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } }
I20260128 15:03:07.320487 22383 raft_consensus.cc:359] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } }
I20260128 15:03:07.320657 22383 raft_consensus.cc:385] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:07.320662 22381 raft_consensus.cc:385] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:07.320724 22381 raft_consensus.cc:740] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d476c4efff04de7aa1c66ae0992c07f, State: Initialized, Role: FOLLOWER
I20260128 15:03:07.320736 22383 raft_consensus.cc:740] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4ced995cca48a8a1c2a1d4d6ae6f8d, State: Initialized, Role: FOLLOWER
I20260128 15:03:07.320807 22382 tablet_bootstrap.cc:492] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19: No bootstrap required, opened a new log
I20260128 15:03:07.320977 22382 ts_tablet_manager.cc:1403] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260128 15:03:07.321141 22383 consensus_queue.cc:260] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } }
I20260128 15:03:07.321151 22381 consensus_queue.cc:260] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } }
I20260128 15:03:07.321729 22353 heartbeater.cc:499] Master 127.13.88.190:41065 was elected leader, sending a full tablet report...
I20260128 15:03:07.321943 22383 ts_tablet_manager.cc:1434] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260128 15:03:07.321946 22381 ts_tablet_manager.cc:1434] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260128 15:03:07.322300 22211 heartbeater.cc:499] Master 127.13.88.190:41065 was elected leader, sending a full tablet report...
I20260128 15:03:07.323659 22382 raft_consensus.cc:359] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } }
I20260128 15:03:07.323849 22382 raft_consensus.cc:385] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:07.323906 22382 raft_consensus.cc:740] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88dca1fa2c84717a0e65c9e2d430d19, State: Initialized, Role: FOLLOWER
I20260128 15:03:07.324306 22382 consensus_queue.cc:260] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } }
I20260128 15:03:07.324915 22069 heartbeater.cc:499] Master 127.13.88.190:41065 was elected leader, sending a full tablet report...
I20260128 15:03:07.325129 22382 ts_tablet_manager.cc:1434] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260128 15:03:07.497530 22354 tablet.cc:2378] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260128 15:03:07.519459 22212 tablet.cc:2378] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260128 15:03:07.524147 22070 tablet.cc:2378] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 15:03:07.622741 22387 raft_consensus.cc:493] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:03:07.622928 22387 raft_consensus.cc:515] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } }
I20260128 15:03:07.624051 22387 leader_election.cc:290] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d88dca1fa2c84717a0e65c9e2d430d19 (127.13.88.129:42207), 0d476c4efff04de7aa1c66ae0992c07f (127.13.88.130:45523)
I20260128 15:03:07.627589 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.624316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:42207 (local address 127.13.88.131:60265)
0128 15:03:07.624466 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:07.624481 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:07.624594 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:07.624861 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:07.624864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:07.624905 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:07.625137 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.625144 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.626513 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.626516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:07.627307 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.627314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.627402 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.627413 (+ 11us) client_negotiation.cc:769] Sending connection context
0128 15:03:07.627454 (+ 41us) client_negotiation.cc:240] Negotiation successful
0128 15:03:07.627508 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"spinlock_wait_cycles":27264}
I20260128 15:03:07.627599 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.624316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:45523 (local address 127.13.88.131:43241)
0128 15:03:07.624796 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:07.624807 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:07.624912 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:07.625128 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:07.625130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:07.625147 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:07.625381 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.625387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.626354 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.626356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:07.627179 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.627188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.627316 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.627338 (+ 22us) client_negotiation.cc:769] Sending connection context
0128 15:03:07.627387 (+ 49us) client_negotiation.cc:240] Negotiation successful
0128 15:03:07.627444 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":146,"spinlock_wait_cycles":7680,"thread_start_us":118,"threads_started":1}
I20260128 15:03:07.628057 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.624404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:43241 (local address 127.13.88.130:45523)
0128 15:03:07.624557 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:07.624562 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:07.624888 (+ 326us) server_negotiation.cc:408] Connection header received
0128 15:03:07.625000 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:07.625002 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:07.625044 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:07.625101 (+ 57us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:07.625485 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.626252 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.627317 (+ 1065us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.627797 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.627838 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:07.627888 (+ 50us) server_negotiation.cc:300] Negotiation successful
0128 15:03:07.627945 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260128 15:03:07.628090 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.624451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:60265 (local address 127.13.88.129:42207)
0128 15:03:07.624609 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:07.624613 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:07.624633 (+ 20us) server_negotiation.cc:408] Connection header received
0128 15:03:07.624694 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:07.624697 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:07.624749 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:07.624817 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:07.625260 (+ 443us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.626414 (+ 1154us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.627431 (+ 1017us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.627871 (+ 440us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.627903 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:07.627945 (+ 42us) server_negotiation.cc:300] Negotiation successful
0128 15:03:07.627995 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260128 15:03:07.628711 22165 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0954cd59361144118c1a2cf0063bdca5" candidate_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d476c4efff04de7aa1c66ae0992c07f" is_pre_election: true
I20260128 15:03:07.628715 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0954cd59361144118c1a2cf0063bdca5" candidate_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" is_pre_election: true
I20260128 15:03:07.629002 22165 raft_consensus.cc:2468] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b4ced995cca48a8a1c2a1d4d6ae6f8d in term 0.
I20260128 15:03:07.629002 22017 raft_consensus.cc:2468] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b4ced995cca48a8a1c2a1d4d6ae6f8d in term 0.
I20260128 15:03:07.629516 22241 leader_election.cc:304] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b4ced995cca48a8a1c2a1d4d6ae6f8d, d88dca1fa2c84717a0e65c9e2d430d19; no voters:
I20260128 15:03:07.629770 22387 raft_consensus.cc:2804] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:03:07.629842 22387 raft_consensus.cc:493] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:03:07.629873 22387 raft_consensus.cc:3060] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:07.630762 22387 raft_consensus.cc:515] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } }
I20260128 15:03:07.631083 22387 leader_election.cc:290] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [CANDIDATE]: Term 1 election: Requested vote from peers d88dca1fa2c84717a0e65c9e2d430d19 (127.13.88.129:42207), 0d476c4efff04de7aa1c66ae0992c07f (127.13.88.130:45523)
I20260128 15:03:07.631415 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0954cd59361144118c1a2cf0063bdca5" candidate_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88dca1fa2c84717a0e65c9e2d430d19"
I20260128 15:03:07.631433 22165 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0954cd59361144118c1a2cf0063bdca5" candidate_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d476c4efff04de7aa1c66ae0992c07f"
I20260128 15:03:07.631523 22017 raft_consensus.cc:3060] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:07.631588 22165 raft_consensus.cc:3060] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:07.632452 22017 raft_consensus.cc:2468] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b4ced995cca48a8a1c2a1d4d6ae6f8d in term 1.
I20260128 15:03:07.632627 22165 raft_consensus.cc:2468] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b4ced995cca48a8a1c2a1d4d6ae6f8d in term 1.
I20260128 15:03:07.632746 22241 leader_election.cc:304] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b4ced995cca48a8a1c2a1d4d6ae6f8d, d88dca1fa2c84717a0e65c9e2d430d19; no voters:
I20260128 15:03:07.633026 22387 raft_consensus.cc:2804] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:03:07.633250 22387 raft_consensus.cc:697] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 LEADER]: Becoming Leader. State: Replica: 4b4ced995cca48a8a1c2a1d4d6ae6f8d, State: Running, Role: LEADER
I20260128 15:03:07.633584 22387 consensus_queue.cc:237] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } }
I20260128 15:03:07.636583 21864 catalog_manager.cc:5654] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b4ced995cca48a8a1c2a1d4d6ae6f8d (127.13.88.131). New cstate: current_term: 1 leader_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } health_report { overall_health: UNKNOWN } } }
I20260128 15:03:07.661674 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.658966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.13.88.131:36433)
0128 15:03:07.659083 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:07.659087 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:07.659099 (+ 12us) server_negotiation.cc:408] Connection header received
0128 15:03:07.659172 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:07.659175 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:07.659228 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:07.659304 (+ 76us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:03:07.659598 (+ 294us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.660113 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.660726 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.660878 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.660937 (+ 59us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:03:07.661380 (+ 443us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:03:07.661473 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:07.661529 (+ 56us) server_negotiation.cc:300] Negotiation successful
0128 15:03:07.661572 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260128 15:03:07.666787 22017 raft_consensus.cc:1275] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 FOLLOWER]: Refusing update from remote peer 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:03:07.666787 22165 raft_consensus.cc:1275] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Refusing update from remote peer 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:03:07.667367 22387 consensus_queue.cc:1048] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:03:07.667537 22391 consensus_queue.cc:1048] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:03:07.674593 22396 mvcc.cc:204] Tried to move back new op lower bound from 7248333159076909056 to 7248333158949675008. Current Snapshot: MvccSnapshot[applied={T|T < 7248333159076909056}]
I20260128 15:03:07.674870 22395 mvcc.cc:204] Tried to move back new op lower bound from 7248333159076909056 to 7248333158949675008. Current Snapshot: MvccSnapshot[applied={T|T < 7248333159076909056}]
I20260128 15:03:07.682050 21864 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:03:07.682178 21864 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:03:07.684373 21864 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2: Sending DeleteTablet for 3 replicas of tablet 0954cd59361144118c1a2cf0063bdca5
I20260128 15:03:07.685171 22003 tablet_service.cc:1552] Processing DeleteTablet for tablet 0954cd59361144118c1a2cf0063bdca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:07 UTC) from {username='oryx', principal='oryx/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:53318
I20260128 15:03:07.685199 22287 tablet_service.cc:1552] Processing DeleteTablet for tablet 0954cd59361144118c1a2cf0063bdca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:07 UTC) from {username='oryx', principal='oryx/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:55210
I20260128 15:03:07.685240 22144 tablet_service.cc:1552] Processing DeleteTablet for tablet 0954cd59361144118c1a2cf0063bdca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:07 UTC) from {username='oryx', principal='oryx/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:50542
I20260128 15:03:07.685676 22405 tablet_replica.cc:333] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19: stopping tablet replica
I20260128 15:03:07.685611 21864 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260128 15:03:07.685928 22405 raft_consensus.cc:2243] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260128 15:03:07.685997 21864 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:03:07.686254 22405 raft_consensus.cc:2272] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:07.686599 22404 tablet_replica.cc:333] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: stopping tablet replica
I20260128 15:03:07.686825 22404 raft_consensus.cc:2243] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 LEADER]: Raft consensus shutting down.
I20260128 15:03:07.687265 22404 raft_consensus.cc:2272] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:07.687582 22405 ts_tablet_manager.cc:1916] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:07.688614 22404 ts_tablet_manager.cc:1916] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:07.690172 22405 ts_tablet_manager.cc:1929] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:03:07.690328 22405 log.cc:1199] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/wal/wals/0954cd59361144118c1a2cf0063bdca5
I20260128 15:03:07.690698 22405 ts_tablet_manager.cc:1950] T 0954cd59361144118c1a2cf0063bdca5 P d88dca1fa2c84717a0e65c9e2d430d19: Deleting consensus metadata
I20260128 15:03:07.691051 22404 ts_tablet_manager.cc:1929] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:03:07.691045 22144 tablet_service.cc:1505] Processing CreateTablet for tablet 0672121501954e12949a52e44f6d60c9 (DEFAULT_TABLE table=test-table [id=631fb635ef8c4fb39f1262bb4535a14d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:07.691200 22404 log.cc:1199] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/wal/wals/0954cd59361144118c1a2cf0063bdca5
I20260128 15:03:07.691306 22144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0672121501954e12949a52e44f6d60c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:07.691531 22404 ts_tablet_manager.cc:1950] T 0954cd59361144118c1a2cf0063bdca5 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Deleting consensus metadata
I20260128 15:03:07.692536 21850 catalog_manager.cc:4985] TS 4b4ced995cca48a8a1c2a1d4d6ae6f8d (127.13.88.131:36433): tablet 0954cd59361144118c1a2cf0063bdca5 (table test-table [id=3f9df442f7054632bed2769a009810f6]) successfully deleted
I20260128 15:03:07.692706 22287 tablet_service.cc:1505] Processing CreateTablet for tablet 0672121501954e12949a52e44f6d60c9 (DEFAULT_TABLE table=test-table [id=631fb635ef8c4fb39f1262bb4535a14d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:07.692924 22287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0672121501954e12949a52e44f6d60c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:07.693492 22381 tablet_bootstrap.cc:492] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f: Bootstrap starting.
I20260128 15:03:07.694466 22381 tablet_bootstrap.cc:654] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:07.694590 22383 tablet_bootstrap.cc:492] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Bootstrap starting.
I20260128 15:03:07.694792 21851 catalog_manager.cc:4985] TS d88dca1fa2c84717a0e65c9e2d430d19 (127.13.88.129:42207): tablet 0954cd59361144118c1a2cf0063bdca5 (table test-table [id=3f9df442f7054632bed2769a009810f6]) successfully deleted
I20260128 15:03:07.695361 22003 tablet_service.cc:1505] Processing CreateTablet for tablet 0672121501954e12949a52e44f6d60c9 (DEFAULT_TABLE table=test-table [id=631fb635ef8c4fb39f1262bb4535a14d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:07.695518 22383 tablet_bootstrap.cc:654] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:07.695596 22003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0672121501954e12949a52e44f6d60c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:07.695889 22406 tablet_replica.cc:333] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f: stopping tablet replica
I20260128 15:03:07.696148 22406 raft_consensus.cc:2243] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:03:07.696491 22406 raft_consensus.cc:2272] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:07.697196 22383 tablet_bootstrap.cc:492] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: No bootstrap required, opened a new log
I20260128 15:03:07.697276 22383 ts_tablet_manager.cc:1403] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 15:03:07.697614 22382 tablet_bootstrap.cc:492] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19: Bootstrap starting.
I20260128 15:03:07.697707 22383 raft_consensus.cc:359] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:07.697825 22383 raft_consensus.cc:385] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:07.697839 22406 ts_tablet_manager.cc:1916] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:07.697858 22383 raft_consensus.cc:740] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4ced995cca48a8a1c2a1d4d6ae6f8d, State: Initialized, Role: FOLLOWER
I20260128 15:03:07.697969 22383 consensus_queue.cc:260] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:07.698218 22383 ts_tablet_manager.cc:1434] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:03:07.698562 22382 tablet_bootstrap.cc:654] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:07.699841 22382 tablet_bootstrap.cc:492] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19: No bootstrap required, opened a new log
I20260128 15:03:07.699913 22382 ts_tablet_manager.cc:1403] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260128 15:03:07.700014 22406 ts_tablet_manager.cc:1929] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:03:07.700089 22406 log.cc:1199] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/wal/wals/0954cd59361144118c1a2cf0063bdca5
I20260128 15:03:07.700384 22406 ts_tablet_manager.cc:1950] T 0954cd59361144118c1a2cf0063bdca5 P 0d476c4efff04de7aa1c66ae0992c07f: Deleting consensus metadata
I20260128 15:03:07.700340 22382 raft_consensus.cc:359] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:07.700470 22382 raft_consensus.cc:385] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:07.700507 22382 raft_consensus.cc:740] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88dca1fa2c84717a0e65c9e2d430d19, State: Initialized, Role: FOLLOWER
I20260128 15:03:07.700626 22382 consensus_queue.cc:260] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:07.700862 22382 ts_tablet_manager.cc:1434] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:03:07.701102 22381 tablet_bootstrap.cc:492] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f: No bootstrap required, opened a new log
I20260128 15:03:07.701196 22381 ts_tablet_manager.cc:1403] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260128 15:03:07.701412 21850 catalog_manager.cc:4985] TS 0d476c4efff04de7aa1c66ae0992c07f (127.13.88.130:45523): tablet 0954cd59361144118c1a2cf0063bdca5 (table test-table [id=3f9df442f7054632bed2769a009810f6]) successfully deleted
I20260128 15:03:07.701607 22381 raft_consensus.cc:359] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:07.701707 22381 raft_consensus.cc:385] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:07.701740 22381 raft_consensus.cc:740] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d476c4efff04de7aa1c66ae0992c07f, State: Initialized, Role: FOLLOWER
I20260128 15:03:07.701844 22381 consensus_queue.cc:260] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:07.702075 22381 ts_tablet_manager.cc:1434] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:03:07.827978 22389 raft_consensus.cc:493] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:03:07.828118 22389 raft_consensus.cc:515] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:07.829262 22389 leader_election.cc:290] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d476c4efff04de7aa1c66ae0992c07f (127.13.88.130:45523), 4b4ced995cca48a8a1c2a1d4d6ae6f8d (127.13.88.131:36433)
I20260128 15:03:07.832819 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.829817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:36433 (local address 127.13.88.129:42593)
0128 15:03:07.830119 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:07.830131 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:07.830221 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:07.830541 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:07.830544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:07.830561 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:07.830789 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.830795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.831557 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.831561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:07.832382 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.832391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.832535 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.832557 (+ 22us) client_negotiation.cc:769] Sending connection context
0128 15:03:07.832606 (+ 49us) client_negotiation.cc:240] Negotiation successful
0128 15:03:07.832660 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":84,"threads_started":1}
I20260128 15:03:07.833240 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.830064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:42593 (local address 127.13.88.131:36433)
0128 15:03:07.830185 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:07.830190 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:07.830203 (+ 13us) server_negotiation.cc:408] Connection header received
0128 15:03:07.830370 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:07.830374 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:07.830430 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:07.830527 (+ 97us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:07.830892 (+ 365us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.831416 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.832526 (+ 1110us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.832968 (+ 442us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.833007 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:07.833056 (+ 49us) server_negotiation.cc:300] Negotiation successful
0128 15:03:07.833121 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260128 15:03:07.833664 22410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.829491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:45523 (local address 127.13.88.129:42443)
0128 15:03:07.829717 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:07.829732 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:07.829826 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:07.830260 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:07.830264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:07.830279 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:07.830519 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.830527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.831611 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.831616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:07.832607 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.832619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.833361 (+ 742us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.833382 (+ 21us) client_negotiation.cc:769] Sending connection context
0128 15:03:07.833466 (+ 84us) client_negotiation.cc:240] Negotiation successful
0128 15:03:07.833520 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":67,"threads_started":1}
I20260128 15:03:07.833707 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.829596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:42443 (local address 127.13.88.130:45523)
0128 15:03:07.829751 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:07.829756 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:07.829778 (+ 22us) server_negotiation.cc:408] Connection header received
0128 15:03:07.830004 (+ 226us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:07.830009 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:07.830069 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:07.830159 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:07.830681 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.831447 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.832758 (+ 1311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.833224 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.833263 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:07.833508 (+ 245us) server_negotiation.cc:300] Negotiation successful
0128 15:03:07.833580 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260128 15:03:07.833734 22307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0672121501954e12949a52e44f6d60c9" candidate_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" is_pre_election: true
I20260128 15:03:07.833942 22307 raft_consensus.cc:2468] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d88dca1fa2c84717a0e65c9e2d430d19 in term 0.
I20260128 15:03:07.834235 22165 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0672121501954e12949a52e44f6d60c9" candidate_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d476c4efff04de7aa1c66ae0992c07f" is_pre_election: true
I20260128 15:03:07.834309 21956 leader_election.cc:304] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b4ced995cca48a8a1c2a1d4d6ae6f8d, d88dca1fa2c84717a0e65c9e2d430d19; no voters:
I20260128 15:03:07.834357 22165 raft_consensus.cc:2468] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d88dca1fa2c84717a0e65c9e2d430d19 in term 0.
I20260128 15:03:07.834560 22389 raft_consensus.cc:2804] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:03:07.834622 22389 raft_consensus.cc:493] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:03:07.834666 22389 raft_consensus.cc:3060] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:07.835520 22389 raft_consensus.cc:515] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:07.835819 22389 leader_election.cc:290] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [CANDIDATE]: Term 1 election: Requested vote from peers 0d476c4efff04de7aa1c66ae0992c07f (127.13.88.130:45523), 4b4ced995cca48a8a1c2a1d4d6ae6f8d (127.13.88.131:36433)
I20260128 15:03:07.836230 22307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0672121501954e12949a52e44f6d60c9" candidate_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d"
I20260128 15:03:07.836230 22165 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0672121501954e12949a52e44f6d60c9" candidate_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d476c4efff04de7aa1c66ae0992c07f"
I20260128 15:03:07.836341 22165 raft_consensus.cc:3060] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:07.836341 22307 raft_consensus.cc:3060] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:07.837352 22165 raft_consensus.cc:2468] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88dca1fa2c84717a0e65c9e2d430d19 in term 1.
I20260128 15:03:07.837352 22307 raft_consensus.cc:2468] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88dca1fa2c84717a0e65c9e2d430d19 in term 1.
I20260128 15:03:07.837613 21956 leader_election.cc:304] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d476c4efff04de7aa1c66ae0992c07f, d88dca1fa2c84717a0e65c9e2d430d19; no voters:
I20260128 15:03:07.837786 22389 raft_consensus.cc:2804] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:03:07.837985 22389 raft_consensus.cc:697] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 LEADER]: Becoming Leader. State: Replica: d88dca1fa2c84717a0e65c9e2d430d19, State: Running, Role: LEADER
I20260128 15:03:07.838225 22389 consensus_queue.cc:237] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:07.840502 21860 catalog_manager.cc:5654] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 reported cstate change: term changed from 0 to 1, leader changed from <none> to d88dca1fa2c84717a0e65c9e2d430d19 (127.13.88.129). New cstate: current_term: 1 leader_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } health_report { overall_health: UNKNOWN } } }
Jan 28 15:03:07 dist-test-slave-m28t krb5kdc[21816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612586, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.190@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM
I20260128 15:03:07.881315 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.874063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:41065 (local address 127.0.0.1:43340)
0128 15:03:07.874369 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:07.874385 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:07.874515 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:07.874750 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:07.874753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:07.875193 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:07.875391 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.875398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.876134 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.876138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:07.876553 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:07.876561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.876660 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.877197 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:07.877210 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:07.878607 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:07.879962 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:07.879970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:07.879975 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:07.880179 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:07.880384 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:07.880387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:07.880389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:07.880453 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:07.880763 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:07.880770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:07.880863 (+ 93us) client_negotiation.cc:769] Sending connection context
0128 15:03:07.880956 (+ 93us) client_negotiation.cc:240] Negotiation successful
0128 15:03:07.881103 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":109,"threads_started":1}
I20260128 15:03:07.881316 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:07.874173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.13.88.190:41065)
0128 15:03:07.874395 (+ 222us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:07.874399 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:07.874463 (+ 64us) server_negotiation.cc:408] Connection header received
0128 15:03:07.874599 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:07.874603 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:07.874641 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:07.874711 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:07.875517 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.876022 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:07.876651 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:07.876814 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:07.878720 (+ 1906us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:07.878735 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:07.878737 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:07.878758 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:07.879862 (+ 1104us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:07.880264 (+ 402us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:07.880268 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:07.880270 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:07.880312 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:07.880527 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:07.880530 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:07.880532 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:07.880665 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:07.880747 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:07.881012 (+ 265us) server_negotiation.cc:300] Negotiation successful
0128 15:03:07.881158 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20260128 15:03:07.885026 21860 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:43340:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260128 15:03:07.885735 21860 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:03:07.890298 22003 tablet_service.cc:1505] Processing CreateTablet for tablet 813dd536e1354f0085756affb50d3d5a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cfda471b64de40e6a1d5b49dbed335af]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 15:03:07.890327 22144 tablet_service.cc:1505] Processing CreateTablet for tablet 813dd536e1354f0085756affb50d3d5a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cfda471b64de40e6a1d5b49dbed335af]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 15:03:07.890690 22144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 813dd536e1354f0085756affb50d3d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:07.890698 22003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 813dd536e1354f0085756affb50d3d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:07.892874 22382 tablet_bootstrap.cc:492] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19: Bootstrap starting.
I20260128 15:03:07.892938 22381 tablet_bootstrap.cc:492] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f: Bootstrap starting.
I20260128 15:03:07.893473 22287 tablet_service.cc:1505] Processing CreateTablet for tablet 813dd536e1354f0085756affb50d3d5a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cfda471b64de40e6a1d5b49dbed335af]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260128 15:03:07.893738 22287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 813dd536e1354f0085756affb50d3d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:07.893898 22382 tablet_bootstrap.cc:654] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:07.893992 22381 tablet_bootstrap.cc:654] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:07.895108 22381 tablet_bootstrap.cc:492] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f: No bootstrap required, opened a new log
I20260128 15:03:07.895198 22381 ts_tablet_manager.cc:1403] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260128 15:03:07.895437 22382 tablet_bootstrap.cc:492] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19: No bootstrap required, opened a new log
I20260128 15:03:07.895519 22382 ts_tablet_manager.cc:1403] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 15:03:07.895679 22381 raft_consensus.cc:359] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:07.895774 22381 raft_consensus.cc:385] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:07.895807 22381 raft_consensus.cc:740] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d476c4efff04de7aa1c66ae0992c07f, State: Initialized, Role: FOLLOWER
I20260128 15:03:07.895921 22381 consensus_queue.cc:260] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:07.895955 22382 raft_consensus.cc:359] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:07.896035 22382 raft_consensus.cc:385] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:07.896049 22383 tablet_bootstrap.cc:492] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Bootstrap starting.
I20260128 15:03:07.896068 22382 raft_consensus.cc:740] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88dca1fa2c84717a0e65c9e2d430d19, State: Initialized, Role: FOLLOWER
I20260128 15:03:07.896195 22381 ts_tablet_manager.cc:1434] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:03:07.896209 22382 consensus_queue.cc:260] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:07.896489 22382 ts_tablet_manager.cc:1434] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:03:07.896507 22388 tablet_replica.cc:442] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } }
I20260128 15:03:07.896617 22388 tablet_replica.cc:445] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:03:07.896925 22383 tablet_bootstrap.cc:654] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:07.897454 22389 tablet_replica.cc:442] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } }
I20260128 15:03:07.897573 22389 tablet_replica.cc:445] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:03:07.898332 22383 tablet_bootstrap.cc:492] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: No bootstrap required, opened a new log
I20260128 15:03:07.898419 22383 ts_tablet_manager.cc:1403] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260128 15:03:07.898815 22383 raft_consensus.cc:359] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:07.898921 22383 raft_consensus.cc:385] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:07.898958 22383 raft_consensus.cc:740] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4ced995cca48a8a1c2a1d4d6ae6f8d, State: Initialized, Role: FOLLOWER
I20260128 15:03:07.899072 22383 consensus_queue.cc:260] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:07.899339 22383 ts_tablet_manager.cc:1434] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:03:07.899529 22400 tablet_replica.cc:442] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } }
I20260128 15:03:07.899641 22400 tablet_replica.cc:445] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:03:08.218817 22400 raft_consensus.cc:493] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:03:08.218968 22400 raft_consensus.cc:515] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.219413 22400 leader_election.cc:290] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d88dca1fa2c84717a0e65c9e2d430d19 (127.13.88.129:42207), 0d476c4efff04de7aa1c66ae0992c07f (127.13.88.130:45523)
I20260128 15:03:08.219920 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813dd536e1354f0085756affb50d3d5a" candidate_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" is_pre_election: true
I20260128 15:03:08.219929 22165 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813dd536e1354f0085756affb50d3d5a" candidate_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d476c4efff04de7aa1c66ae0992c07f" is_pre_election: true
I20260128 15:03:08.220063 22017 raft_consensus.cc:2468] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b4ced995cca48a8a1c2a1d4d6ae6f8d in term 0.
I20260128 15:03:08.220075 22165 raft_consensus.cc:2468] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b4ced995cca48a8a1c2a1d4d6ae6f8d in term 0.
I20260128 15:03:08.220363 22240 leader_election.cc:304] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d476c4efff04de7aa1c66ae0992c07f, 4b4ced995cca48a8a1c2a1d4d6ae6f8d; no voters:
I20260128 15:03:08.220559 22400 raft_consensus.cc:2804] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:03:08.220623 22400 raft_consensus.cc:493] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:03:08.220650 22400 raft_consensus.cc:3060] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:08.221566 22400 raft_consensus.cc:515] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.221915 22400 leader_election.cc:290] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [CANDIDATE]: Term 1 election: Requested vote from peers d88dca1fa2c84717a0e65c9e2d430d19 (127.13.88.129:42207), 0d476c4efff04de7aa1c66ae0992c07f (127.13.88.130:45523)
I20260128 15:03:08.222273 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813dd536e1354f0085756affb50d3d5a" candidate_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88dca1fa2c84717a0e65c9e2d430d19"
I20260128 15:03:08.222275 22165 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "813dd536e1354f0085756affb50d3d5a" candidate_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d476c4efff04de7aa1c66ae0992c07f"
I20260128 15:03:08.222395 22165 raft_consensus.cc:3060] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:08.222397 22017 raft_consensus.cc:3060] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:08.223172 22017 raft_consensus.cc:2468] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b4ced995cca48a8a1c2a1d4d6ae6f8d in term 1.
I20260128 15:03:08.223208 22165 raft_consensus.cc:2468] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b4ced995cca48a8a1c2a1d4d6ae6f8d in term 1.
I20260128 15:03:08.223449 22241 leader_election.cc:304] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b4ced995cca48a8a1c2a1d4d6ae6f8d, d88dca1fa2c84717a0e65c9e2d430d19; no voters:
I20260128 15:03:08.223616 22400 raft_consensus.cc:2804] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:03:08.223778 22400 raft_consensus.cc:697] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 LEADER]: Becoming Leader. State: Replica: 4b4ced995cca48a8a1c2a1d4d6ae6f8d, State: Running, Role: LEADER
I20260128 15:03:08.223904 22400 consensus_queue.cc:237] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.224555 22424 tablet_replica.cc:442] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: TxnStatusTablet state changed. Reason: New leader 4b4ced995cca48a8a1c2a1d4d6ae6f8d. Latest consensus state: current_term: 1 leader_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } }
I20260128 15:03:08.224658 22424 tablet_replica.cc:445] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: This TxnStatusTablet replica's current role is: LEADER
I20260128 15:03:08.224962 22426 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260128 15:03:08.225099 22426 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260128 15:03:08.225576 21860 catalog_manager.cc:5654] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b4ced995cca48a8a1c2a1d4d6ae6f8d (127.13.88.131). New cstate: current_term: 1 leader_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } health_report { overall_health: HEALTHY } } }
I20260128 15:03:08.258754 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:08.255422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.13.88.131:36433)
0128 15:03:08.255547 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:08.255552 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:08.255667 (+ 115us) server_negotiation.cc:408] Connection header received
0128 15:03:08.255858 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:08.255861 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:08.255916 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:08.255989 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:03:08.256409 (+ 420us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.257175 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:08.258066 (+ 891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.258251 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:08.258292 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:03:08.258375 (+ 83us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:03:08.258450 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:08.258563 (+ 113us) server_negotiation.cc:300] Negotiation successful
0128 15:03:08.258620 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260128 15:03:08.258780 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:08.255334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:36433 (local address 127.0.0.1:55242)
0128 15:03:08.255591 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:08.255606 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:08.255735 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:08.256009 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:08.256012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:08.256020 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 15:03:08.256288 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:08.256295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.257282 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:08.257286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:08.257955 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:08.257965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.258078 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:08.258132 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 15:03:08.258464 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 15:03:08.258478 (+ 14us) client_negotiation.cc:769] Sending connection context
0128 15:03:08.258548 (+ 70us) client_negotiation.cc:240] Negotiation successful
0128 15:03:08.258640 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260128 15:03:08.261962 22165 raft_consensus.cc:1275] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Refusing update from remote peer 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:03:08.261957 22017 raft_consensus.cc:1275] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 FOLLOWER]: Refusing update from remote peer 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:03:08.262483 22424 consensus_queue.cc:1048] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:03:08.262646 22400 consensus_queue.cc:1048] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:03:08.264001 22389 tablet_replica.cc:442] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19: TxnStatusTablet state changed. Reason: New leader 4b4ced995cca48a8a1c2a1d4d6ae6f8d. Latest consensus state: current_term: 1 leader_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } }
I20260128 15:03:08.264103 22389 tablet_replica.cc:445] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:03:08.264164 22388 tablet_replica.cc:442] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f: TxnStatusTablet state changed. Reason: New leader 4b4ced995cca48a8a1c2a1d4d6ae6f8d. Latest consensus state: current_term: 1 leader_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } }
I20260128 15:03:08.264271 22388 tablet_replica.cc:445] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:03:08.266454 22429 mvcc.cc:204] Tried to move back new op lower bound from 7248333161516294144 to 7248333161366343680. Current Snapshot: MvccSnapshot[applied={T|T < 7248333161516294144}]
I20260128 15:03:08.266752 22424 tablet_replica.cc:442] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } }
I20260128 15:03:08.266834 22424 tablet_replica.cc:445] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: This TxnStatusTablet replica's current role is: LEADER
I20260128 15:03:08.266731 22400 tablet_replica.cc:442] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } }
I20260128 15:03:08.267161 22400 tablet_replica.cc:445] T 813dd536e1354f0085756affb50d3d5a P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: This TxnStatusTablet replica's current role is: LEADER
I20260128 15:03:08.267372 22388 tablet_replica.cc:442] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } }
I20260128 15:03:08.267462 22388 tablet_replica.cc:445] T 813dd536e1354f0085756affb50d3d5a P 0d476c4efff04de7aa1c66ae0992c07f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:03:08.267891 22389 tablet_replica.cc:442] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } } }
I20260128 15:03:08.267982 22389 tablet_replica.cc:445] T 813dd536e1354f0085756affb50d3d5a P d88dca1fa2c84717a0e65c9e2d430d19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260128 15:03:08.276438 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:08.272951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.13.88.129:42207)
0128 15:03:08.273392 (+ 441us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:08.273395 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:08.273405 (+ 10us) server_negotiation.cc:408] Connection header received
0128 15:03:08.273470 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:08.273472 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:08.273519 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:08.273607 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:03:08.274028 (+ 421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.274515 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:08.275400 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.275577 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:08.275640 (+ 63us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:03:08.276072 (+ 432us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:03:08.276188 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:08.276258 (+ 70us) server_negotiation.cc:300] Negotiation successful
0128 15:03:08.276301 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":62,"threads_started":1}
I20260128 15:03:08.285959 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:08.282905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:36433 (local address 127.13.88.129:43659)
0128 15:03:08.283245 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:08.283256 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:08.283339 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:08.283563 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:08.283565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:08.283571 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 15:03:08.283774 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:08.283778 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.284767 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:08.284772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:08.285221 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:08.285230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.285328 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:08.285380 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 15:03:08.285732 (+ 352us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 15:03:08.285735 (+ 3us) client_negotiation.cc:769] Sending connection context
0128 15:03:08.285777 (+ 42us) client_negotiation.cc:240] Negotiation successful
0128 15:03:08.285838 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":74,"threads_started":1}
I20260128 15:03:08.285972 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:08.283009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:43659 (local address 127.13.88.131:36433)
0128 15:03:08.283127 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:08.283131 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:08.283318 (+ 187us) server_negotiation.cc:408] Connection header received
0128 15:03:08.283417 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:08.283420 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:08.283473 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:08.283560 (+ 87us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:03:08.283858 (+ 298us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.284632 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:08.285328 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.285529 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:08.285577 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:03:08.285655 (+ 78us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:03:08.285730 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:08.285803 (+ 73us) server_negotiation.cc:300] Negotiation successful
0128 15:03:08.285848 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260128 15:03:08.292730 22165 raft_consensus.cc:1275] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Refusing update from remote peer d88dca1fa2c84717a0e65c9e2d430d19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:03:08.293012 22307 raft_consensus.cc:1275] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Refusing update from remote peer d88dca1fa2c84717a0e65c9e2d430d19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:03:08.293160 22412 consensus_queue.cc:1048] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:03:08.293466 22389 consensus_queue.cc:1048] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:03:08.315025 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:08.311864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:42207 (local address 127.13.88.131:42225)
0128 15:03:08.312151 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:08.312182 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:08.312266 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:08.312530 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:08.312534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:08.312542 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 15:03:08.312773 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:08.312783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.313782 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:08.313785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:08.314257 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:08.314266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.314352 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:08.314387 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 15:03:08.314816 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 15:03:08.314819 (+ 3us) client_negotiation.cc:769] Sending connection context
0128 15:03:08.314861 (+ 42us) client_negotiation.cc:240] Negotiation successful
0128 15:03:08.314915 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":98,"threads_started":1}
I20260128 15:03:08.315065 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:08.311950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:42225 (local address 127.13.88.129:42207)
0128 15:03:08.312072 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:08.312077 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:08.312242 (+ 165us) server_negotiation.cc:408] Connection header received
0128 15:03:08.312362 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:08.312365 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:08.312447 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:08.312512 (+ 65us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:03:08.312877 (+ 365us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.313679 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:08.314363 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.314606 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:08.314655 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:03:08.314737 (+ 82us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:03:08.314812 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:08.314888 (+ 76us) server_negotiation.cc:300] Negotiation successful
0128 15:03:08.314940 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260128 15:03:08.341753 21863 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:03:08.341923 21863 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:03:08.344306 21863 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2: Sending DeleteTablet for 3 replicas of tablet 0672121501954e12949a52e44f6d60c9
I20260128 15:03:08.344851 22003 tablet_service.cc:1552] Processing DeleteTablet for tablet 0672121501954e12949a52e44f6d60c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:08 UTC) from {username='oryx', principal='oryx/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:53318
I20260128 15:03:08.345068 22144 tablet_service.cc:1552] Processing DeleteTablet for tablet 0672121501954e12949a52e44f6d60c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:08 UTC) from {username='oryx', principal='oryx/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:50542
I20260128 15:03:08.345345 22287 tablet_service.cc:1552] Processing DeleteTablet for tablet 0672121501954e12949a52e44f6d60c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:08 UTC) from {username='oryx', principal='oryx/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:55210
I20260128 15:03:08.345472 22451 tablet_replica.cc:333] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f: stopping tablet replica
I20260128 15:03:08.345595 22451 raft_consensus.cc:2243] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:03:08.345764 22451 raft_consensus.cc:2272] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:08.346500 22451 ts_tablet_manager.cc:1916] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:08.349406 22451 ts_tablet_manager.cc:1929] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 15:03:08.349552 22451 log.cc:1199] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/wal/wals/0672121501954e12949a52e44f6d60c9
I20260128 15:03:08.349871 22451 ts_tablet_manager.cc:1950] T 0672121501954e12949a52e44f6d60c9 P 0d476c4efff04de7aa1c66ae0992c07f: Deleting consensus metadata
I20260128 15:03:08.350277 22450 tablet_replica.cc:333] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19: stopping tablet replica
I20260128 15:03:08.350521 22452 tablet_replica.cc:333] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: stopping tablet replica
I20260128 15:03:08.350657 21850 catalog_manager.cc:4985] TS 0d476c4efff04de7aa1c66ae0992c07f (127.13.88.130:45523): tablet 0672121501954e12949a52e44f6d60c9 (table test-table [id=631fb635ef8c4fb39f1262bb4535a14d]) successfully deleted
I20260128 15:03:08.350692 22450 raft_consensus.cc:2243] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 LEADER]: Raft consensus shutting down.
I20260128 15:03:08.350986 22450 raft_consensus.cc:2272] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:08.351178 22452 raft_consensus.cc:2243] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:03:08.351349 22452 raft_consensus.cc:2272] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:08.352105 22452 ts_tablet_manager.cc:1916] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:08.352859 22450 ts_tablet_manager.cc:1916] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:08.355515 22452 ts_tablet_manager.cc:1929] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 15:03:08.355643 22452 log.cc:1199] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/wal/wals/0672121501954e12949a52e44f6d60c9
I20260128 15:03:08.355967 22452 ts_tablet_manager.cc:1950] T 0672121501954e12949a52e44f6d60c9 P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Deleting consensus metadata
I20260128 15:03:08.356813 21850 catalog_manager.cc:4985] TS 4b4ced995cca48a8a1c2a1d4d6ae6f8d (127.13.88.131:36433): tablet 0672121501954e12949a52e44f6d60c9 (table test-table [id=631fb635ef8c4fb39f1262bb4535a14d]) successfully deleted
I20260128 15:03:08.357666 22450 ts_tablet_manager.cc:1929] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 15:03:08.357731 22450 log.cc:1199] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/wal/wals/0672121501954e12949a52e44f6d60c9
I20260128 15:03:08.357924 22450 ts_tablet_manager.cc:1950] T 0672121501954e12949a52e44f6d60c9 P d88dca1fa2c84717a0e65c9e2d430d19: Deleting consensus metadata
I20260128 15:03:08.358601 21851 catalog_manager.cc:4985] TS d88dca1fa2c84717a0e65c9e2d430d19 (127.13.88.129:42207): tablet 0672121501954e12949a52e44f6d60c9 (table test-table [id=631fb635ef8c4fb39f1262bb4535a14d]) successfully deleted
Jan 28 15:03:08 dist-test-slave-m28t krb5kdc[21816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jan 28 15:03:08 dist-test-slave-m28t krb5kdc[21816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:08 dist-test-slave-m28t krb5kdc[21816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
W20260128 15:03:08.374115 22420 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:08.368868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.13.88.190:41065)
0128 15:03:08.369023 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:08.369029 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:08.369149 (+ 120us) server_negotiation.cc:408] Connection header received
0128 15:03:08.369281 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:08.369285 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:08.369333 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:08.369396 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:08.370177 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.370693 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:08.371307 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.371512 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:08.373994 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43344: BlockingRecv error: recv got EOF from 127.0.0.1:43344 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jan 28 15:03:08 dist-test-slave-m28t krb5kdc[21816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM
I20260128 15:03:08.382477 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:08.375437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.13.88.190:41065)
0128 15:03:08.375550 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:08.375555 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:08.375950 (+ 395us) server_negotiation.cc:408] Connection header received
0128 15:03:08.376050 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:08.376054 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:08.376094 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:08.376178 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:08.376854 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.377316 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:08.377872 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.378304 (+ 432us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:08.379864 (+ 1560us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:08.379882 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:08.379885 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:08.379914 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:08.381207 (+ 1293us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:08.381545 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:08.381548 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:08.381550 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:08.381590 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:08.381773 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:08.381776 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:08.381778 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:08.381915 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:08.382028 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:08.382213 (+ 185us) server_negotiation.cc:300] Negotiation successful
0128 15:03:08.382320 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260128 15:03:08.384727 21863 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 15:03:08.385311 21863 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:03:08.390120 22003 tablet_service.cc:1505] Processing CreateTablet for tablet e56fb5571bca4f03bd5d79ed2ec8ab9c (DEFAULT_TABLE table=test-table [id=b4583062221941e499533e702cb25352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:08.390393 22003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e56fb5571bca4f03bd5d79ed2ec8ab9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:08.392554 22382 tablet_bootstrap.cc:492] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19: Bootstrap starting.
I20260128 15:03:08.393302 22382 tablet_bootstrap.cc:654] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:08.393951 22144 tablet_service.cc:1505] Processing CreateTablet for tablet e56fb5571bca4f03bd5d79ed2ec8ab9c (DEFAULT_TABLE table=test-table [id=b4583062221941e499533e702cb25352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:08.394174 22144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e56fb5571bca4f03bd5d79ed2ec8ab9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:08.396083 22381 tablet_bootstrap.cc:492] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f: Bootstrap starting.
I20260128 15:03:08.396461 22287 tablet_service.cc:1505] Processing CreateTablet for tablet e56fb5571bca4f03bd5d79ed2ec8ab9c (DEFAULT_TABLE table=test-table [id=b4583062221941e499533e702cb25352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:08.396720 22287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e56fb5571bca4f03bd5d79ed2ec8ab9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:08.397032 22381 tablet_bootstrap.cc:654] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:08.398801 22383 tablet_bootstrap.cc:492] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Bootstrap starting.
I20260128 15:03:08.399639 22383 tablet_bootstrap.cc:654] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:08.400126 22382 tablet_bootstrap.cc:492] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19: No bootstrap required, opened a new log
I20260128 15:03:08.400216 22382 ts_tablet_manager.cc:1403] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260128 15:03:08.400727 22382 raft_consensus.cc:359] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.400851 22382 raft_consensus.cc:385] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:08.400884 22382 raft_consensus.cc:740] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88dca1fa2c84717a0e65c9e2d430d19, State: Initialized, Role: FOLLOWER
I20260128 15:03:08.401005 22382 consensus_queue.cc:260] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.401270 22383 tablet_bootstrap.cc:492] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: No bootstrap required, opened a new log
I20260128 15:03:08.401283 22382 ts_tablet_manager.cc:1434] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:03:08.401340 22383 ts_tablet_manager.cc:1403] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 15:03:08.401782 22383 raft_consensus.cc:359] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.401878 22383 raft_consensus.cc:385] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:08.401909 22383 raft_consensus.cc:740] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4ced995cca48a8a1c2a1d4d6ae6f8d, State: Initialized, Role: FOLLOWER
I20260128 15:03:08.402020 22383 consensus_queue.cc:260] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.402257 22383 ts_tablet_manager.cc:1434] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:03:08.402717 22381 tablet_bootstrap.cc:492] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f: No bootstrap required, opened a new log
I20260128 15:03:08.402794 22381 ts_tablet_manager.cc:1403] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260128 15:03:08.403228 22381 raft_consensus.cc:359] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.403331 22381 raft_consensus.cc:385] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:08.403357 22381 raft_consensus.cc:740] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d476c4efff04de7aa1c66ae0992c07f, State: Initialized, Role: FOLLOWER
I20260128 15:03:08.403486 22381 consensus_queue.cc:260] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.403744 22381 ts_tablet_manager.cc:1434] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:03:08.746698 22412 raft_consensus.cc:493] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:03:08.746846 22412 raft_consensus.cc:515] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.747288 22412 leader_election.cc:290] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d476c4efff04de7aa1c66ae0992c07f (127.13.88.130:45523), 4b4ced995cca48a8a1c2a1d4d6ae6f8d (127.13.88.131:36433)
I20260128 15:03:08.747812 22164 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e56fb5571bca4f03bd5d79ed2ec8ab9c" candidate_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d476c4efff04de7aa1c66ae0992c07f" is_pre_election: true
I20260128 15:03:08.747859 22307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e56fb5571bca4f03bd5d79ed2ec8ab9c" candidate_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" is_pre_election: true
I20260128 15:03:08.747972 22164 raft_consensus.cc:2468] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d88dca1fa2c84717a0e65c9e2d430d19 in term 0.
I20260128 15:03:08.748032 22307 raft_consensus.cc:2468] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d88dca1fa2c84717a0e65c9e2d430d19 in term 0.
I20260128 15:03:08.748353 21956 leader_election.cc:304] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b4ced995cca48a8a1c2a1d4d6ae6f8d, d88dca1fa2c84717a0e65c9e2d430d19; no voters:
I20260128 15:03:08.748605 22412 raft_consensus.cc:2804] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:03:08.748682 22412 raft_consensus.cc:493] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:03:08.748732 22412 raft_consensus.cc:3060] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:08.749727 22412 raft_consensus.cc:515] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.750084 22412 leader_election.cc:290] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [CANDIDATE]: Term 1 election: Requested vote from peers 0d476c4efff04de7aa1c66ae0992c07f (127.13.88.130:45523), 4b4ced995cca48a8a1c2a1d4d6ae6f8d (127.13.88.131:36433)
I20260128 15:03:08.750507 22307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e56fb5571bca4f03bd5d79ed2ec8ab9c" candidate_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d"
I20260128 15:03:08.750540 22164 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e56fb5571bca4f03bd5d79ed2ec8ab9c" candidate_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d476c4efff04de7aa1c66ae0992c07f"
I20260128 15:03:08.750626 22307 raft_consensus.cc:3060] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:08.750648 22164 raft_consensus.cc:3060] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:08.751528 22307 raft_consensus.cc:2468] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88dca1fa2c84717a0e65c9e2d430d19 in term 1.
I20260128 15:03:08.751623 22164 raft_consensus.cc:2468] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d88dca1fa2c84717a0e65c9e2d430d19 in term 1.
I20260128 15:03:08.751988 21956 leader_election.cc:304] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d476c4efff04de7aa1c66ae0992c07f, d88dca1fa2c84717a0e65c9e2d430d19; no voters:
I20260128 15:03:08.752167 22412 raft_consensus.cc:2804] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:03:08.752266 22412 raft_consensus.cc:697] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 LEADER]: Becoming Leader. State: Replica: d88dca1fa2c84717a0e65c9e2d430d19, State: Running, Role: LEADER
I20260128 15:03:08.752455 22412 consensus_queue.cc:237] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.754051 21860 catalog_manager.cc:5654] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 reported cstate change: term changed from 0 to 1, leader changed from <none> to d88dca1fa2c84717a0e65c9e2d430d19 (127.13.88.129). New cstate: current_term: 1 leader_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } health_report { overall_health: UNKNOWN } } }
I20260128 15:03:08.847523 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:08.844926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53332 (local address 127.13.88.129:42207)
0128 15:03:08.845183 (+ 257us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:08.845187 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:08.845200 (+ 13us) server_negotiation.cc:408] Connection header received
0128 15:03:08.845246 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:08.845249 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:08.845297 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:08.845371 (+ 74us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:03:08.845801 (+ 430us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.846321 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:08.846922 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:08.847057 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:08.847096 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:03:08.847190 (+ 94us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:03:08.847249 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:08.847321 (+ 72us) server_negotiation.cc:300] Negotiation successful
0128 15:03:08.847364 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":94,"threads_started":1}
I20260128 15:03:08.849875 22307 raft_consensus.cc:1275] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Refusing update from remote peer d88dca1fa2c84717a0e65c9e2d430d19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:03:08.849875 22164 raft_consensus.cc:1275] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Refusing update from remote peer d88dca1fa2c84717a0e65c9e2d430d19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:03:08.850276 22412 consensus_queue.cc:1048] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:03:08.850440 22389 consensus_queue.cc:1048] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:03:08.857683 21864 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:03:08.857837 21864 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:03:08.859617 21864 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2: Sending DeleteTablet for 3 replicas of tablet e56fb5571bca4f03bd5d79ed2ec8ab9c
I20260128 15:03:08.860085 22003 tablet_service.cc:1552] Processing DeleteTablet for tablet e56fb5571bca4f03bd5d79ed2ec8ab9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:08 UTC) from {username='oryx', principal='oryx/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:53318
I20260128 15:03:08.860105 22144 tablet_service.cc:1552] Processing DeleteTablet for tablet e56fb5571bca4f03bd5d79ed2ec8ab9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:08 UTC) from {username='oryx', principal='oryx/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:50542
I20260128 15:03:08.860126 22287 tablet_service.cc:1552] Processing DeleteTablet for tablet e56fb5571bca4f03bd5d79ed2ec8ab9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:08 UTC) from {username='oryx', principal='oryx/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:55210
I20260128 15:03:08.860499 22479 tablet_replica.cc:333] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f: stopping tablet replica
I20260128 15:03:08.860627 22479 raft_consensus.cc:2243] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:03:08.860761 22479 raft_consensus.cc:2272] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:08.860806 22481 tablet_replica.cc:333] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19: stopping tablet replica
I20260128 15:03:08.860965 22480 tablet_replica.cc:333] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: stopping tablet replica
I20260128 15:03:08.860889 21864 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260128 15:03:08.861012 22481 raft_consensus.cc:2243] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 LEADER]: Raft consensus shutting down.
I20260128 15:03:08.861191 22481 raft_consensus.cc:2272] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:08.861251 22480 raft_consensus.cc:2243] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260128 15:03:08.861243 21864 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:03:08.861369 22480 raft_consensus.cc:2272] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:08.861785 22481 ts_tablet_manager.cc:1916] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:08.862046 22480 ts_tablet_manager.cc:1916] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:08.864398 22480 ts_tablet_manager.cc:1929] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:03:08.864398 22481 ts_tablet_manager.cc:1929] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:03:08.864502 22481 log.cc:1199] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-0/wal/wals/e56fb5571bca4f03bd5d79ed2ec8ab9c
I20260128 15:03:08.864502 22480 log.cc:1199] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/wal/wals/e56fb5571bca4f03bd5d79ed2ec8ab9c
I20260128 15:03:08.864760 22480 ts_tablet_manager.cc:1950] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Deleting consensus metadata
I20260128 15:03:08.864760 22481 ts_tablet_manager.cc:1950] T e56fb5571bca4f03bd5d79ed2ec8ab9c P d88dca1fa2c84717a0e65c9e2d430d19: Deleting consensus metadata
I20260128 15:03:08.864866 22479 ts_tablet_manager.cc:1916] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:08.865342 21851 catalog_manager.cc:4985] TS d88dca1fa2c84717a0e65c9e2d430d19 (127.13.88.129:42207): tablet e56fb5571bca4f03bd5d79ed2ec8ab9c (table test-table [id=b4583062221941e499533e702cb25352]) successfully deleted
I20260128 15:03:08.865517 21850 catalog_manager.cc:4985] TS 4b4ced995cca48a8a1c2a1d4d6ae6f8d (127.13.88.131:36433): tablet e56fb5571bca4f03bd5d79ed2ec8ab9c (table test-table [id=b4583062221941e499533e702cb25352]) successfully deleted
I20260128 15:03:08.866361 22287 tablet_service.cc:1505] Processing CreateTablet for tablet 2052e3f9a86c44c9b2f101bd99f499ae (DEFAULT_TABLE table=test-table [id=e0377509b5b946eab8dbc3181d4a7722]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:08.866597 22287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2052e3f9a86c44c9b2f101bd99f499ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:08.866554 22003 tablet_service.cc:1505] Processing CreateTablet for tablet 2052e3f9a86c44c9b2f101bd99f499ae (DEFAULT_TABLE table=test-table [id=e0377509b5b946eab8dbc3181d4a7722]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:08.866555 22144 tablet_service.cc:1505] Processing CreateTablet for tablet 2052e3f9a86c44c9b2f101bd99f499ae (DEFAULT_TABLE table=test-table [id=e0377509b5b946eab8dbc3181d4a7722]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:08.866755 22003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2052e3f9a86c44c9b2f101bd99f499ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:08.866794 22144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2052e3f9a86c44c9b2f101bd99f499ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:08.867260 22479 ts_tablet_manager.cc:1929] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:03:08.867318 22479 log.cc:1199] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/wal/wals/e56fb5571bca4f03bd5d79ed2ec8ab9c
I20260128 15:03:08.867660 22479 ts_tablet_manager.cc:1950] T e56fb5571bca4f03bd5d79ed2ec8ab9c P 0d476c4efff04de7aa1c66ae0992c07f: Deleting consensus metadata
I20260128 15:03:08.868779 22382 tablet_bootstrap.cc:492] T 2052e3f9a86c44c9b2f101bd99f499ae P d88dca1fa2c84717a0e65c9e2d430d19: Bootstrap starting.
I20260128 15:03:08.868824 22381 tablet_bootstrap.cc:492] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f: Bootstrap starting.
I20260128 15:03:08.868883 21850 catalog_manager.cc:4985] TS 0d476c4efff04de7aa1c66ae0992c07f (127.13.88.130:45523): tablet e56fb5571bca4f03bd5d79ed2ec8ab9c (table test-table [id=b4583062221941e499533e702cb25352]) successfully deleted
I20260128 15:03:08.869671 22382 tablet_bootstrap.cc:654] T 2052e3f9a86c44c9b2f101bd99f499ae P d88dca1fa2c84717a0e65c9e2d430d19: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:08.869781 22381 tablet_bootstrap.cc:654] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:08.870239 22383 tablet_bootstrap.cc:492] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Bootstrap starting.
I20260128 15:03:08.870950 22381 tablet_bootstrap.cc:492] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f: No bootstrap required, opened a new log
I20260128 15:03:08.871039 22381 ts_tablet_manager.cc:1403] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260128 15:03:08.871099 22383 tablet_bootstrap.cc:654] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:08.871516 22381 raft_consensus.cc:359] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.871626 22381 raft_consensus.cc:385] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:08.871665 22381 raft_consensus.cc:740] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d476c4efff04de7aa1c66ae0992c07f, State: Initialized, Role: FOLLOWER
I20260128 15:03:08.871785 22381 consensus_queue.cc:260] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.871873 22382 tablet_bootstrap.cc:492] T 2052e3f9a86c44c9b2f101bd99f499ae P d88dca1fa2c84717a0e65c9e2d430d19: No bootstrap required, opened a new log
I20260128 15:03:08.871950 22382 ts_tablet_manager.cc:1403] T 2052e3f9a86c44c9b2f101bd99f499ae P d88dca1fa2c84717a0e65c9e2d430d19: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260128 15:03:08.872046 22381 ts_tablet_manager.cc:1434] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:03:08.872334 22382 raft_consensus.cc:359] T 2052e3f9a86c44c9b2f101bd99f499ae P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.872484 22382 raft_consensus.cc:385] T 2052e3f9a86c44c9b2f101bd99f499ae P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:08.872527 22382 raft_consensus.cc:740] T 2052e3f9a86c44c9b2f101bd99f499ae P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88dca1fa2c84717a0e65c9e2d430d19, State: Initialized, Role: FOLLOWER
I20260128 15:03:08.872637 22382 consensus_queue.cc:260] T 2052e3f9a86c44c9b2f101bd99f499ae P d88dca1fa2c84717a0e65c9e2d430d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.872869 22383 tablet_bootstrap.cc:492] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: No bootstrap required, opened a new log
I20260128 15:03:08.872885 22382 ts_tablet_manager.cc:1434] T 2052e3f9a86c44c9b2f101bd99f499ae P d88dca1fa2c84717a0e65c9e2d430d19: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:03:08.872946 22383 ts_tablet_manager.cc:1403] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260128 15:03:08.873363 22383 raft_consensus.cc:359] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.873458 22383 raft_consensus.cc:385] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:08.873490 22383 raft_consensus.cc:740] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4ced995cca48a8a1c2a1d4d6ae6f8d, State: Initialized, Role: FOLLOWER
I20260128 15:03:08.873605 22383 consensus_queue.cc:260] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:08.873822 22383 ts_tablet_manager.cc:1434] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260128 15:03:09.214258 22388 raft_consensus.cc:493] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:03:09.214407 22388 raft_consensus.cc:515] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:09.215488 22388 leader_election.cc:290] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d88dca1fa2c84717a0e65c9e2d430d19 (127.13.88.129:42207), 4b4ced995cca48a8a1c2a1d4d6ae6f8d (127.13.88.131:36433)
I20260128 15:03:09.218470 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:09.215741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:36433 (local address 127.13.88.130:50159)
0128 15:03:09.216058 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:09.216069 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:09.216222 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:09.216499 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:09.216502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:09.216528 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:09.216744 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:09.216751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.217463 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:09.217466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:09.218042 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:09.218049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.218147 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:09.218169 (+ 22us) client_negotiation.cc:769] Sending connection context
0128 15:03:09.218222 (+ 53us) client_negotiation.cc:240] Negotiation successful
0128 15:03:09.218279 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"spinlock_wait_cycles":50688,"thread_start_us":73,"threads_started":1}
I20260128 15:03:09.218490 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:09.215726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:42207 (local address 127.13.88.130:40191)
0128 15:03:09.216051 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:09.216064 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:09.216225 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:09.216497 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:09.216500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:09.216523 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:09.216760 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:09.216766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.217456 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:09.217460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:09.218036 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:09.218044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.218150 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:09.218169 (+ 19us) client_negotiation.cc:769] Sending connection context
0128 15:03:09.218224 (+ 55us) client_negotiation.cc:240] Negotiation successful
0128 15:03:09.218285 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"spinlock_wait_cycles":7296,"thread_start_us":114,"threads_started":1}
I20260128 15:03:09.218890 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:09.215827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:50159 (local address 127.13.88.131:36433)
0128 15:03:09.216051 (+ 224us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:09.216055 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:09.216174 (+ 119us) server_negotiation.cc:408] Connection header received
0128 15:03:09.216329 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:09.216332 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:09.216377 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:09.216488 (+ 111us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:09.216856 (+ 368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.217352 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:09.218161 (+ 809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.218617 (+ 456us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:09.218653 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:09.218705 (+ 52us) server_negotiation.cc:300] Negotiation successful
0128 15:03:09.218780 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":53,"threads_started":1}
I20260128 15:03:09.218902 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:09.215798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:40191 (local address 127.13.88.129:42207)
0128 15:03:09.215928 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:09.215932 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:09.216158 (+ 226us) server_negotiation.cc:408] Connection header received
0128 15:03:09.216337 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:09.216340 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:09.216386 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:09.216485 (+ 99us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:09.216871 (+ 386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.217352 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:09.218150 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.218631 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:09.218671 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:09.218722 (+ 51us) server_negotiation.cc:300] Negotiation successful
0128 15:03:09.218791 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260128 15:03:09.219260 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2052e3f9a86c44c9b2f101bd99f499ae" candidate_uuid: "0d476c4efff04de7aa1c66ae0992c07f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" is_pre_election: true
I20260128 15:03:09.219259 22307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2052e3f9a86c44c9b2f101bd99f499ae" candidate_uuid: "0d476c4efff04de7aa1c66ae0992c07f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" is_pre_election: true
I20260128 15:03:09.219401 22017 raft_consensus.cc:2468] T 2052e3f9a86c44c9b2f101bd99f499ae P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d476c4efff04de7aa1c66ae0992c07f in term 0.
I20260128 15:03:09.219401 22307 raft_consensus.cc:2468] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d476c4efff04de7aa1c66ae0992c07f in term 0.
I20260128 15:03:09.219784 22099 leader_election.cc:304] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d476c4efff04de7aa1c66ae0992c07f, d88dca1fa2c84717a0e65c9e2d430d19; no voters:
I20260128 15:03:09.220014 22388 raft_consensus.cc:2804] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:03:09.220073 22388 raft_consensus.cc:493] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:03:09.220110 22388 raft_consensus.cc:3060] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:09.221063 22388 raft_consensus.cc:515] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:09.221400 22388 leader_election.cc:290] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [CANDIDATE]: Term 1 election: Requested vote from peers d88dca1fa2c84717a0e65c9e2d430d19 (127.13.88.129:42207), 4b4ced995cca48a8a1c2a1d4d6ae6f8d (127.13.88.131:36433)
I20260128 15:03:09.221791 22017 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2052e3f9a86c44c9b2f101bd99f499ae" candidate_uuid: "0d476c4efff04de7aa1c66ae0992c07f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d88dca1fa2c84717a0e65c9e2d430d19"
I20260128 15:03:09.221791 22307 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2052e3f9a86c44c9b2f101bd99f499ae" candidate_uuid: "0d476c4efff04de7aa1c66ae0992c07f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d"
I20260128 15:03:09.221918 22307 raft_consensus.cc:3060] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:09.221918 22017 raft_consensus.cc:3060] T 2052e3f9a86c44c9b2f101bd99f499ae P d88dca1fa2c84717a0e65c9e2d430d19 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:09.222965 22017 raft_consensus.cc:2468] T 2052e3f9a86c44c9b2f101bd99f499ae P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d476c4efff04de7aa1c66ae0992c07f in term 1.
I20260128 15:03:09.222965 22307 raft_consensus.cc:2468] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d476c4efff04de7aa1c66ae0992c07f in term 1.
I20260128 15:03:09.223296 22098 leader_election.cc:304] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d476c4efff04de7aa1c66ae0992c07f, 4b4ced995cca48a8a1c2a1d4d6ae6f8d; no voters:
I20260128 15:03:09.223539 22388 raft_consensus.cc:2804] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:03:09.223737 22388 raft_consensus.cc:697] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [term 1 LEADER]: Becoming Leader. State: Replica: 0d476c4efff04de7aa1c66ae0992c07f, State: Running, Role: LEADER
I20260128 15:03:09.223991 22388 consensus_queue.cc:237] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } }
I20260128 15:03:09.226352 21863 catalog_manager.cc:5654] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d476c4efff04de7aa1c66ae0992c07f (127.13.88.130). New cstate: current_term: 1 leader_uuid: "0d476c4efff04de7aa1c66ae0992c07f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d476c4efff04de7aa1c66ae0992c07f" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 45523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 } health_report { overall_health: UNKNOWN } } }
I20260128 15:03:09.246905 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:09.243902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.13.88.130:45523)
0128 15:03:09.244145 (+ 243us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:09.244149 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:09.244163 (+ 14us) server_negotiation.cc:408] Connection header received
0128 15:03:09.244232 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:09.244235 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:09.244284 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:09.244373 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:03:09.244669 (+ 296us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.245179 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:09.245893 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.246033 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:09.246082 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:03:09.246564 (+ 482us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:03:09.246650 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:09.246721 (+ 71us) server_negotiation.cc:300] Negotiation successful
0128 15:03:09.246764 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":101,"threads_started":1}
I20260128 15:03:09.256265 22495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:09.253478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:36433 (local address 127.13.88.130:51341)
0128 15:03:09.253831 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:09.253843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:09.253947 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:09.254167 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:09.254170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:09.254195 (+ 25us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 15:03:09.254419 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:09.254425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.255091 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:09.255095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:09.255528 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:09.255536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.255632 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:09.255674 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 15:03:09.255989 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 15:03:09.255995 (+ 6us) client_negotiation.cc:769] Sending connection context
0128 15:03:09.256050 (+ 55us) client_negotiation.cc:240] Negotiation successful
0128 15:03:09.256120 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":82,"threads_started":1}
I20260128 15:03:09.256250 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:09.253600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:51341 (local address 127.13.88.131:36433)
0128 15:03:09.253711 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:09.253714 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:09.253917 (+ 203us) server_negotiation.cc:408] Connection header received
0128 15:03:09.254040 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:09.254043 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:09.254089 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:09.254149 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:03:09.254527 (+ 378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.254994 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:09.255633 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.255805 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:09.255844 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:03:09.255909 (+ 65us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:03:09.255968 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:09.256068 (+ 100us) server_negotiation.cc:300] Negotiation successful
0128 15:03:09.256111 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260128 15:03:09.262157 22017 raft_consensus.cc:1275] T 2052e3f9a86c44c9b2f101bd99f499ae P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 FOLLOWER]: Refusing update from remote peer 0d476c4efff04de7aa1c66ae0992c07f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:03:09.262202 22307 raft_consensus.cc:1275] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Refusing update from remote peer 0d476c4efff04de7aa1c66ae0992c07f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:03:09.262650 22488 consensus_queue.cc:1048] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b4ced995cca48a8a1c2a1d4d6ae6f8d" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 36433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:03:09.262810 22388 consensus_queue.cc:1048] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d88dca1fa2c84717a0e65c9e2d430d19" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 42207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:03:09.278170 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:09.275115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:48047 (local address 127.13.88.130:45523)
0128 15:03:09.275236 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:09.275241 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:09.275389 (+ 148us) server_negotiation.cc:408] Connection header received
0128 15:03:09.275514 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:09.275518 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:09.275565 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:09.275625 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:03:09.275971 (+ 346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.276927 (+ 956us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:09.277549 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.277733 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:09.277772 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:03:09.277834 (+ 62us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:03:09.277895 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:09.277989 (+ 94us) server_negotiation.cc:300] Negotiation successful
0128 15:03:09.278032 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260128 15:03:09.278169 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:09.275079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:45523 (local address 127.13.88.131:48047)
0128 15:03:09.275319 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:09.275332 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:09.275430 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:09.275646 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:09.275649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:09.275657 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0128 15:03:09.275874 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:09.275879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.277031 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:09.277034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:09.277433 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:09.277440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:09.277555 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:09.277586 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0128 15:03:09.277913 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0128 15:03:09.277919 (+ 6us) client_negotiation.cc:769] Sending connection context
0128 15:03:09.277973 (+ 54us) client_negotiation.cc:240] Negotiation successful
0128 15:03:09.278043 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":95,"threads_started":1}
I20260128 15:03:09.300956 21860 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:03:09.301141 21860 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:03:09.303362 21860 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 24a668aa02bd496e84c8cace1b6f5bf2: Sending DeleteTablet for 3 replicas of tablet 2052e3f9a86c44c9b2f101bd99f499ae
I20260128 15:03:09.303931 22003 tablet_service.cc:1552] Processing DeleteTablet for tablet 2052e3f9a86c44c9b2f101bd99f499ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:09 UTC) from {username='oryx', principal='oryx/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:53318
I20260128 15:03:09.303990 22144 tablet_service.cc:1552] Processing DeleteTablet for tablet 2052e3f9a86c44c9b2f101bd99f499ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:09 UTC) from {username='oryx', principal='oryx/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:50542
I20260128 15:03:09.304126 22481 tablet_replica.cc:333] T 2052e3f9a86c44c9b2f101bd99f499ae P d88dca1fa2c84717a0e65c9e2d430d19: stopping tablet replica
I20260128 15:03:09.304159 22479 tablet_replica.cc:333] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f: stopping tablet replica
I20260128 15:03:09.304255 22481 raft_consensus.cc:2243] T 2052e3f9a86c44c9b2f101bd99f499ae P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:03:09.304258 22479 raft_consensus.cc:2243] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [term 1 LEADER]: Raft consensus shutting down.
I20260128 15:03:09.304361 22481 raft_consensus.cc:2272] T 2052e3f9a86c44c9b2f101bd99f499ae P d88dca1fa2c84717a0e65c9e2d430d19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:09.304595 22479 raft_consensus.cc:2272] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:09.304649 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 21938
I20260128 15:03:09.305622 22287 tablet_service.cc:1552] Processing DeleteTablet for tablet 2052e3f9a86c44c9b2f101bd99f499ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:09 UTC) from {username='oryx', principal='oryx/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:55210
I20260128 15:03:09.305791 22480 tablet_replica.cc:333] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: stopping tablet replica
I20260128 15:03:09.305869 22479 ts_tablet_manager.cc:1916] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:09.305884 22480 raft_consensus.cc:2243] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:03:09.305992 22480 raft_consensus.cc:2272] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:09.308233 22479 ts_tablet_manager.cc:1929] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 15:03:09.308306 22479 log.cc:1199] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-1/wal/wals/2052e3f9a86c44c9b2f101bd99f499ae
I20260128 15:03:09.308838 22479 ts_tablet_manager.cc:1950] T 2052e3f9a86c44c9b2f101bd99f499ae P 0d476c4efff04de7aa1c66ae0992c07f: Deleting consensus metadata
I20260128 15:03:09.309543 22480 ts_tablet_manager.cc:1916] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:09.309547 21850 catalog_manager.cc:4985] TS 0d476c4efff04de7aa1c66ae0992c07f (127.13.88.130:45523): tablet 2052e3f9a86c44c9b2f101bd99f499ae (table test-table [id=e0377509b5b946eab8dbc3181d4a7722]) successfully deleted
I20260128 15:03:09.311997 22480 ts_tablet_manager.cc:1929] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260128 15:03:09.312079 22480 log.cc:1199] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1769612519049359-13666-0/minicluster-data/ts-2/wal/wals/2052e3f9a86c44c9b2f101bd99f499ae
W20260128 15:03:09.312731 21851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.88.129:42207 (error 108)
I20260128 15:03:09.312991 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 22080
I20260128 15:03:09.313226 22480 ts_tablet_manager.cc:1950] T 2052e3f9a86c44c9b2f101bd99f499ae P 4b4ced995cca48a8a1c2a1d4d6ae6f8d: Deleting consensus metadata
I20260128 15:03:09.314329 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:09.313387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:42207 (local address 127.0.0.1:53348)
0128 15:03:09.314160 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:09.314229 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.129:42207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":48,"threads_started":1}
I20260128 15:03:09.314384 21850 catalog_manager.cc:4985] TS 4b4ced995cca48a8a1c2a1d4d6ae6f8d (127.13.88.131:36433): tablet 2052e3f9a86c44c9b2f101bd99f499ae (table test-table [id=e0377509b5b946eab8dbc3181d4a7722]) successfully deleted
W20260128 15:03:09.314533 21851 catalog_manager.cc:4712] TS d88dca1fa2c84717a0e65c9e2d430d19 (127.13.88.129:42207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2052e3f9a86c44c9b2f101bd99f499ae: Network error: Client connection negotiation failed: client connection to 127.13.88.129:42207: connect: Connection refused (error 111)
I20260128 15:03:09.318663 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 22222
I20260128 15:03:09.324568 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 21832
2026-01-28T15:03:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4955 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:03:09 dist-test-slave-m28t krb5kdc[22509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:03:09 dist-test-slave-m28t krb5kdc[22509](info): set up 2 sockets
Jan 28 15:03:09 dist-test-slave-m28t krb5kdc[22509](info): commencing operation
krb5kdc: starting...
W20260128 15:03:11.160787 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.814s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Jan 28 15:03:11 dist-test-slave-m28t krb5kdc[22509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:03:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:03:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:03:11.293020 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:35543
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:35543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:03:11.398550 22525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:11.398803 22525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:11.398880 22525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:11.402262 22525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:03:11.402339 22525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:11.402370 22525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:11.402395 22525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:03:11.402427 22525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:03:11.406817 22525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:35543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:35543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:11.407908 22525 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:11.408805 22525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:11.414206 22531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:11.414209 22530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:11.414320 22525 server_base.cc:1060] running on GCE node
W20260128 15:03:11.414212 22533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:11.414999 22525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:11.415908 22525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:11.417052 22525 hybrid_clock.cc:648] HybridClock initialized: now 1769612591417027 us; error 40 us; skew 500 ppm
W20260128 15:03:11.420131 22525 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-01-28T15:03:11Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2094 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:03:11 dist-test-slave-m28t krb5kdc[22538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:03:11 dist-test-slave-m28t krb5kdc[22538](info): set up 2 sockets
Jan 28 15:03:11 dist-test-slave-m28t krb5kdc[22538](info): commencing operation
krb5kdc: starting...
W20260128 15:03:13.257196 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.815s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Jan 28 15:03:13 dist-test-slave-m28t krb5kdc[22538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:03:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:03:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:03:13.391373 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:39313
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:39313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:03:13.502991 22554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:13.503243 22554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:13.503309 22554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:13.506711 22554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:03:13.506789 22554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:13.506814 22554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:13.506847 22554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:03:13.506882 22554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:03:13.511380 22554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:39313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:39313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:13.512513 22554 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:13.513401 22554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:13.518980 22559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:13.518980 22560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:13.518980 22562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:13.519096 22554 server_base.cc:1060] running on GCE node
I20260128 15:03:13.519707 22554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:13.520677 22554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:13.521823 22554 hybrid_clock.cc:648] HybridClock initialized: now 1769612593521808 us; error 41 us; skew 500 ppm
Jan 28 15:03:13 dist-test-slave-m28t krb5kdc[22538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612593, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:13.524344 22554 init.cc:377] Logged in from keytab as kudu/127.13.88.190@KRBTEST.COM (short username kudu)
I20260128 15:03:13.525499 22554 webserver.cc:492] Webserver started at http://127.13.88.190:46313/ using document root <none> and password file <none>
I20260128 15:03:13.526043 22554 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:13.526095 22554 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:13.526278 22554 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:13.527981 22554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "8a0214a26b2f4eab8246d49936b74222"
format_stamp: "Formatted at 2026-01-28 15:03:13 on dist-test-slave-m28t"
server_key: "f5506ef95d7633aed161176e70a904e3"
server_key_iv: "63c8177abc58e7b7bb61b2064b3d9e04"
server_key_version: "encryptionkey@0"
I20260128 15:03:13.528499 22554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "8a0214a26b2f4eab8246d49936b74222"
format_stamp: "Formatted at 2026-01-28 15:03:13 on dist-test-slave-m28t"
server_key: "f5506ef95d7633aed161176e70a904e3"
server_key_iv: "63c8177abc58e7b7bb61b2064b3d9e04"
server_key_version: "encryptionkey@0"
I20260128 15:03:13.531795 22554 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260128 15:03:13.534090 22569 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:13.535070 22554 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:03:13.535180 22554 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "8a0214a26b2f4eab8246d49936b74222"
format_stamp: "Formatted at 2026-01-28 15:03:13 on dist-test-slave-m28t"
server_key: "f5506ef95d7633aed161176e70a904e3"
server_key_iv: "63c8177abc58e7b7bb61b2064b3d9e04"
server_key_version: "encryptionkey@0"
I20260128 15:03:13.535267 22554 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:13.552354 22554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:13.554903 22554 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:13.555068 22554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:13.562285 22554 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:39313
I20260128 15:03:13.562353 22621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:39313 every 8 connection(s)
I20260128 15:03:13.563295 22554 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:03:13.566028 22622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:13.566847 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 22554
I20260128 15:03:13.566977 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:03:13.567265 13666 external_mini_cluster.cc:1468] Setting key df7a44d3775c1984fb4b3d445a832ec9
I20260128 15:03:13.572681 22622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222: Bootstrap starting.
Jan 28 15:03:13 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:03:13.575260 22622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:13.575978 22622 log.cc:826] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222: Log is configured to *not* fsync() on all Append() calls
I20260128 15:03:13.577998 22622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222: No bootstrap required, opened a new log
I20260128 15:03:13.579473 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:13.568504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.13.88.190:39313)
0128 15:03:13.568877 (+ 373us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:13.568886 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:13.568922 (+ 36us) server_negotiation.cc:408] Connection header received
0128 15:03:13.569476 (+ 554us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:13.569494 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:13.569782 (+ 288us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:13.570055 (+ 273us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:13.571092 (+ 1037us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:13.571820 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:13.572696 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:13.572959 (+ 263us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:13.575328 (+ 2369us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:13.575346 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:13.575359 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:13.575407 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:13.577071 (+ 1664us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:13.577519 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:13.577523 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:13.577527 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:13.577597 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:13.577815 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:13.577817 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:13.577818 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:13.578177 (+ 359us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:13.578336 (+ 159us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:13.578608 (+ 272us) server_negotiation.cc:300] Negotiation successful
0128 15:03:13.578854 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":103,"threads_started":1}
I20260128 15:03:13.580735 22622 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0214a26b2f4eab8246d49936b74222" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39313 } }
I20260128 15:03:13.580929 22622 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:13.580997 22622 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a0214a26b2f4eab8246d49936b74222, State: Initialized, Role: FOLLOWER
I20260128 15:03:13.581533 22622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0214a26b2f4eab8246d49936b74222" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39313 } }
I20260128 15:03:13.581705 22622 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:03:13.581774 22622 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:03:13.581863 22622 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:13.582738 22622 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0214a26b2f4eab8246d49936b74222" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39313 } }
I20260128 15:03:13.583065 22622 leader_election.cc:304] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a0214a26b2f4eab8246d49936b74222; no voters:
I20260128 15:03:13.583341 22622 leader_election.cc:290] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:03:13.583453 22627 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:03:13.583648 22627 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 1 LEADER]: Becoming Leader. State: Replica: 8a0214a26b2f4eab8246d49936b74222, State: Running, Role: LEADER
I20260128 15:03:13.583948 22627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0214a26b2f4eab8246d49936b74222" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39313 } }
I20260128 15:03:13.584393 22622 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:03:13.585474 22628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a0214a26b2f4eab8246d49936b74222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0214a26b2f4eab8246d49936b74222" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39313 } } }
I20260128 15:03:13.585615 22628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:13.585963 22636 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:03:13.585932 22629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a0214a26b2f4eab8246d49936b74222. Latest consensus state: current_term: 1 leader_uuid: "8a0214a26b2f4eab8246d49936b74222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0214a26b2f4eab8246d49936b74222" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39313 } } }
I20260128 15:03:13.586153 22629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:13.589100 22636 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:03:13.594565 22636 catalog_manager.cc:1357] Generated new cluster ID: 4188ab20ec7c4a44a3ac26b9401692c0
I20260128 15:03:13.594659 22636 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:03:13.612560 22636 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:03:13.613559 22636 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:03:13.622563 22636 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222: Generated new TSK 0
I20260128 15:03:13.623435 22636 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:03:13.675058 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:39313
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:03:13.782691 22650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:13.782931 22650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:13.782996 22650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:13.786314 22650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:13.786381 22650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:13.786479 22650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:03:13.790758 22650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:39313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:13.791795 22650 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:13.792673 22650 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:13.799103 22655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:13.799170 22656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:13.799264 22658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:13.799293 22650 server_base.cc:1060] running on GCE node
I20260128 15:03:13.799654 22650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:13.800180 22650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:13.801319 22650 hybrid_clock.cc:648] HybridClock initialized: now 1769612593801295 us; error 40 us; skew 500 ppm
Jan 28 15:03:13 dist-test-slave-m28t krb5kdc[22538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612593, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:13.803700 22650 init.cc:377] Logged in from keytab as kudu/127.13.88.129@KRBTEST.COM (short username kudu)
I20260128 15:03:13.804754 22650 webserver.cc:492] Webserver started at http://127.13.88.129:40907/ using document root <none> and password file <none>
I20260128 15:03:13.805289 22650 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:13.805338 22650 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:13.805509 22650 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:13.807168 22650 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "5f6e324bf3e348eeb32f0fc1d47d25c8"
format_stamp: "Formatted at 2026-01-28 15:03:13 on dist-test-slave-m28t"
server_key: "a25c00a35769f9621e95c1fda47c45e1"
server_key_iv: "680682a989a7256f60e01a07a6927025"
server_key_version: "encryptionkey@0"
I20260128 15:03:13.807623 22650 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "5f6e324bf3e348eeb32f0fc1d47d25c8"
format_stamp: "Formatted at 2026-01-28 15:03:13 on dist-test-slave-m28t"
server_key: "a25c00a35769f9621e95c1fda47c45e1"
server_key_iv: "680682a989a7256f60e01a07a6927025"
server_key_version: "encryptionkey@0"
I20260128 15:03:13.810834 22650 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260128 15:03:13.812907 22665 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:13.813911 22650 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:03:13.814011 22650 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "5f6e324bf3e348eeb32f0fc1d47d25c8"
format_stamp: "Formatted at 2026-01-28 15:03:13 on dist-test-slave-m28t"
server_key: "a25c00a35769f9621e95c1fda47c45e1"
server_key_iv: "680682a989a7256f60e01a07a6927025"
server_key_version: "encryptionkey@0"
I20260128 15:03:13.814093 22650 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:13.828078 22650 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:13.830613 22650 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:13.830790 22650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:13.831331 22650 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:13.832185 22650 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:13.832233 22650 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:13.832293 22650 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:13.832319 22650 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:13.840752 22650 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:39845
I20260128 15:03:13.840780 22778 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:39845 every 8 connection(s)
I20260128 15:03:13.841655 22650 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
Jan 28 15:03:13 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612593, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:03:13.850423 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 22650
I20260128 15:03:13.850566 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:03:13.850847 13666 external_mini_cluster.cc:1468] Setting key 88762a897d43d34834bfebd78e566fcb
I20260128 15:03:13.853111 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:13.843340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:53351 (local address 127.13.88.190:39313)
0128 15:03:13.843490 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:13.843496 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:13.844148 (+ 652us) server_negotiation.cc:408] Connection header received
0128 15:03:13.844950 (+ 802us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:13.844953 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:13.845001 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:13.845072 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:13.846648 (+ 1576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:13.847137 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:13.847716 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:13.847838 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:13.849839 (+ 2001us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:13.849859 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:13.849861 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:13.849884 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:13.851278 (+ 1394us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:13.851740 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:13.851742 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:13.851744 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:13.851786 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:13.852057 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:13.852060 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:13.852062 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:13.852222 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:13.852335 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:13.852832 (+ 497us) server_negotiation.cc:300] Negotiation successful
0128 15:03:13.852959 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260128 15:03:13.853669 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:13.843600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:53351)
0128 15:03:13.844009 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:13.844045 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:13.844781 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:13.845207 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:13.845214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:13.845607 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:13.846509 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:13.846523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:13.847223 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:13.847226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:13.847636 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:13.847642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:13.847807 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:13.848335 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:13.848359 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:13.849718 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:13.851384 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:13.851389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:13.851397 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:13.851655 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:13.851862 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:13.851865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:13.851866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:13.851977 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:13.852335 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:13.852340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:13.852604 (+ 264us) client_negotiation.cc:769] Sending connection context
0128 15:03:13.852771 (+ 167us) client_negotiation.cc:240] Negotiation successful
0128 15:03:13.852978 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":101,"threads_started":1}
I20260128 15:03:13.854784 22779 heartbeater.cc:344] Connected to a master server at 127.13.88.190:39313
I20260128 15:03:13.855007 22779 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:13.855512 22779 heartbeater.cc:507] Master 127.13.88.190:39313 requested a full tablet report, sending...
I20260128 15:03:13.856966 22586 ts_manager.cc:194] Registered new tserver with Master: 5f6e324bf3e348eeb32f0fc1d47d25c8 (127.13.88.129:39845)
I20260128 15:03:13.858107 22586 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.129@KRBTEST.COM'} at 127.13.88.129:53351
WARNING: no policy specified for kudu/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:03:13.893209 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:39313
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:03:13.998936 22786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:13.999150 22786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:13.999207 22786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:14.002491 22786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:14.002556 22786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:14.002635 22786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:03:14.006860 22786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:39313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:14.007962 22786 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:14.008803 22786 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:14.015240 22791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:14.015300 22794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:14.015471 22786 server_base.cc:1060] running on GCE node
W20260128 15:03:14.015240 22792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:14.016058 22786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:14.016636 22786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:14.017776 22786 hybrid_clock.cc:648] HybridClock initialized: now 1769612594017762 us; error 30 us; skew 500 ppm
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612594, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:14.020386 22786 init.cc:377] Logged in from keytab as kudu/127.13.88.130@KRBTEST.COM (short username kudu)
I20260128 15:03:14.021519 22786 webserver.cc:492] Webserver started at http://127.13.88.130:35969/ using document root <none> and password file <none>
I20260128 15:03:14.022037 22786 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:14.022084 22786 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:14.022267 22786 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:14.023968 22786 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "2215a7809b60430bb4d619b3baa9fdb1"
format_stamp: "Formatted at 2026-01-28 15:03:14 on dist-test-slave-m28t"
server_key: "08d4670981029d15983afe4337282002"
server_key_iv: "a438ba1e88aa87e02e3c8b36f3b5f18e"
server_key_version: "encryptionkey@0"
I20260128 15:03:14.024452 22786 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "2215a7809b60430bb4d619b3baa9fdb1"
format_stamp: "Formatted at 2026-01-28 15:03:14 on dist-test-slave-m28t"
server_key: "08d4670981029d15983afe4337282002"
server_key_iv: "a438ba1e88aa87e02e3c8b36f3b5f18e"
server_key_version: "encryptionkey@0"
I20260128 15:03:14.027912 22786 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260128 15:03:14.030165 22801 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:14.031250 22786 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:03:14.031353 22786 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "2215a7809b60430bb4d619b3baa9fdb1"
format_stamp: "Formatted at 2026-01-28 15:03:14 on dist-test-slave-m28t"
server_key: "08d4670981029d15983afe4337282002"
server_key_iv: "a438ba1e88aa87e02e3c8b36f3b5f18e"
server_key_version: "encryptionkey@0"
I20260128 15:03:14.031453 22786 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:14.060226 22786 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:14.062935 22786 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:14.063108 22786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:14.063687 22786 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:14.064595 22786 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:14.064642 22786 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:14.064687 22786 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:14.064715 22786 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:14.073272 22786 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:35191
I20260128 15:03:14.073313 22914 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:35191 every 8 connection(s)
I20260128 15:03:14.074215 22786 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
I20260128 15:03:14.078748 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 22786
I20260128 15:03:14.078863 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:03:14.079114 13666 external_mini_cluster.cc:1468] Setting key 22fe4d23ab28b73fb210d4691d020a28
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612594, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:03:14.086999 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:14.076082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:39221 (local address 127.13.88.190:39313)
0128 15:03:14.076238 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:14.076243 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:14.076966 (+ 723us) server_negotiation.cc:408] Connection header received
0128 15:03:14.077771 (+ 805us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:14.077775 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:14.077823 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:14.077896 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:14.079636 (+ 1740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.080187 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:14.080917 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.081094 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:14.083419 (+ 2325us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:14.083437 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:14.083439 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:14.083465 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:14.084812 (+ 1347us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:14.085483 (+ 671us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:14.085486 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:14.085488 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:14.085542 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:14.085834 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:14.085854 (+ 20us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:14.085856 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:14.086072 (+ 216us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:14.086149 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:14.086770 (+ 621us) server_negotiation.cc:300] Negotiation successful
0128 15:03:14.086871 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260128 15:03:14.087918 22917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:14.076390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:39221)
0128 15:03:14.076839 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:14.076874 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:14.077598 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:14.078043 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:14.078051 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:14.078457 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:14.079455 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:14.079476 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.080322 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:14.080325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:14.080801 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:14.080810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.080989 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:14.081559 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:14.081594 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:14.083264 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:14.084928 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:14.084935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:14.084950 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:14.085380 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:14.085620 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:14.085623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:14.085625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:14.085756 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:14.086165 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:14.086173 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:14.086501 (+ 328us) client_negotiation.cc:769] Sending connection context
0128 15:03:14.086760 (+ 259us) client_negotiation.cc:240] Negotiation successful
0128 15:03:14.087057 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.13.88.131@KRBTEST.COM; defaulting to no policy
I20260128 15:03:14.089044 22915 heartbeater.cc:344] Connected to a master server at 127.13.88.190:39313
I20260128 15:03:14.089264 22915 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:14.089795 22915 heartbeater.cc:507] Master 127.13.88.190:39313 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.131@KRBTEST.COM" created.
I20260128 15:03:14.090844 22586 ts_manager.cc:194] Registered new tserver with Master: 2215a7809b60430bb4d619b3baa9fdb1 (127.13.88.130:35191)
I20260128 15:03:14.091528 22586 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.130@KRBTEST.COM'} at 127.13.88.130:39221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:03:14.125151 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:39313
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:03:14.230952 22922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:14.231182 22922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:14.231240 22922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:14.234472 22922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:14.234540 22922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:14.234624 22922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:03:14.238912 22922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:39313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:14.239991 22922 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:14.240805 22922 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:14.246989 22928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:14.247018 22927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:14.247195 22930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:14.247376 22922 server_base.cc:1060] running on GCE node
I20260128 15:03:14.247714 22922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:14.248263 22922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:14.249409 22922 hybrid_clock.cc:648] HybridClock initialized: now 1769612594249393 us; error 33 us; skew 500 ppm
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612594, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:14.251961 22922 init.cc:377] Logged in from keytab as kudu/127.13.88.131@KRBTEST.COM (short username kudu)
I20260128 15:03:14.253010 22922 webserver.cc:492] Webserver started at http://127.13.88.131:36821/ using document root <none> and password file <none>
I20260128 15:03:14.253532 22922 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:14.253579 22922 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:14.253755 22922 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:14.255435 22922 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "872260e365b240189919f7f8b6867bbe"
format_stamp: "Formatted at 2026-01-28 15:03:14 on dist-test-slave-m28t"
server_key: "c070bfe141ca550a5aa047be32cf5d40"
server_key_iv: "6eff377076f73b809e283db301de1b53"
server_key_version: "encryptionkey@0"
I20260128 15:03:14.255889 22922 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "872260e365b240189919f7f8b6867bbe"
format_stamp: "Formatted at 2026-01-28 15:03:14 on dist-test-slave-m28t"
server_key: "c070bfe141ca550a5aa047be32cf5d40"
server_key_iv: "6eff377076f73b809e283db301de1b53"
server_key_version: "encryptionkey@0"
I20260128 15:03:14.259210 22922 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260128 15:03:14.261380 22937 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:14.262365 22922 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260128 15:03:14.262470 22922 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "872260e365b240189919f7f8b6867bbe"
format_stamp: "Formatted at 2026-01-28 15:03:14 on dist-test-slave-m28t"
server_key: "c070bfe141ca550a5aa047be32cf5d40"
server_key_iv: "6eff377076f73b809e283db301de1b53"
server_key_version: "encryptionkey@0"
I20260128 15:03:14.262562 22922 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:14.278409 22922 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:14.281361 22922 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:14.281551 22922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:14.282091 22922 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:14.282989 22922 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:14.283035 22922 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:14.283087 22922 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:14.283115 22922 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:14.291754 22922 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:35731
I20260128 15:03:14.291782 23050 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:35731 every 8 connection(s)
I20260128 15:03:14.292742 22922 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
I20260128 15:03:14.300494 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 22922
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612594, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:03:14.300611 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:03:14.300843 13666 external_mini_cluster.cc:1468] Setting key ea5a95cb6be07f20708a6d9418e5776a
I20260128 15:03:14.304188 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:14.294355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:54841 (local address 127.13.88.190:39313)
0128 15:03:14.294478 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:14.294482 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:14.295148 (+ 666us) server_negotiation.cc:408] Connection header received
0128 15:03:14.295947 (+ 799us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:14.295951 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:14.296003 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:14.296074 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:14.297836 (+ 1762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.298347 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:14.298924 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.299044 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:14.301128 (+ 2084us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:14.301148 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:14.301150 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:14.301175 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:14.302411 (+ 1236us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:14.302920 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:14.302923 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:14.302925 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:14.302968 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:14.303213 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:14.303217 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:14.303219 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:14.303364 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:14.303447 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:14.303916 (+ 469us) server_negotiation.cc:300] Negotiation successful
0128 15:03:14.304034 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260128 15:03:14.304826 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:14.294597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:54841)
0128 15:03:14.295003 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:14.295040 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:14.295762 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:14.296233 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:14.296240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:14.296765 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:14.297693 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:14.297708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.298445 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:14.298448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:14.298844 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:14.298850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.299022 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:14.299546 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:14.299573 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:14.300987 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:14.302516 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:14.302522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:14.302534 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:14.302825 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:14.303039 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:14.303042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:14.303044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:14.303136 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:14.303455 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:14.303462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:14.303692 (+ 230us) client_negotiation.cc:769] Sending connection context
0128 15:03:14.303872 (+ 180us) client_negotiation.cc:240] Negotiation successful
0128 15:03:14.304127 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":107,"threads_started":1}
I20260128 15:03:14.305723 23051 heartbeater.cc:344] Connected to a master server at 127.13.88.190:39313
I20260128 15:03:14.305945 23051 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:14.306434 23051 heartbeater.cc:507] Master 127.13.88.190:39313 requested a full tablet report, sending...
I20260128 15:03:14.307343 22586 ts_manager.cc:194] Registered new tserver with Master: 872260e365b240189919f7f8b6867bbe (127.13.88.131:35731)
I20260128 15:03:14.307884 22586 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.131@KRBTEST.COM'} at 127.13.88.131:54841
I20260128 15:03:14.314222 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/oryx_127.13.88.129.keytab.
Entry for principal oryx/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/oryx_127.13.88.129.keytab.
WARNING: no policy specified for oryx/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/oryx_127.13.88.130.keytab.
Entry for principal oryx/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/oryx_127.13.88.130.keytab.
WARNING: no policy specified for oryx/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/oryx_127.13.88.131.keytab.
Entry for principal oryx/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/oryx_127.13.88.131.keytab.
I20260128 15:03:14.374868 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 22650
I20260128 15:03:14.379794 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 22786
I20260128 15:03:14.384377 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 22922
I20260128 15:03:14.388911 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 22554
I20260128 15:03:14.394373 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:39313
--webserver_interface=127.13.88.190
--webserver_port=46313
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:39313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:03:14.499830 23060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:14.500074 23060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:14.500150 23060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:14.503481 23060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:03:14.503566 23060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:14.503599 23060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:14.503623 23060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:03:14.503657 23060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:03:14.507978 23060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:39313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:39313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=46313
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:14.509200 23060 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:14.510049 23060 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:14.515597 23065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:14.515614 23068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:14.515597 23066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:14.515913 23060 server_base.cc:1060] running on GCE node
I20260128 15:03:14.516458 23060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:14.517416 23060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:14.518556 23060 hybrid_clock.cc:648] HybridClock initialized: now 1769612594518539 us; error 36 us; skew 500 ppm
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612594, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:14.521165 23060 init.cc:377] Logged in from keytab as kudu/127.13.88.190@KRBTEST.COM (short username kudu)
I20260128 15:03:14.522238 23060 webserver.cc:492] Webserver started at http://127.13.88.190:46313/ using document root <none> and password file <none>
I20260128 15:03:14.522789 23060 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:14.522847 23060 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:14.526647 23060 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260128 15:03:14.528713 23075 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:14.529775 23060 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260128 15:03:14.529877 23060 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "8a0214a26b2f4eab8246d49936b74222"
format_stamp: "Formatted at 2026-01-28 15:03:13 on dist-test-slave-m28t"
server_key: "f5506ef95d7633aed161176e70a904e3"
server_key_iv: "63c8177abc58e7b7bb61b2064b3d9e04"
server_key_version: "encryptionkey@0"
I20260128 15:03:14.530303 23060 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:14.559001 23060 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:14.562074 23060 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:14.562229 23060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:14.569245 23127 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:39313 every 8 connection(s)
I20260128 15:03:14.569248 23060 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:39313
I20260128 15:03:14.570432 23060 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:03:14.575055 23128 sys_catalog.cc:263] Verifying existing consensus state
I20260128 15:03:14.576805 23128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222: Bootstrap starting.
I20260128 15:03:14.579914 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 23060
I20260128 15:03:14.580358 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:39845
--local_ip_for_outbound_sockets=127.13.88.129
--tserver_master_addrs=127.13.88.190:39313
--webserver_port=40907
--webserver_interface=127.13.88.129
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.88.129
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/oryx_127.13.88.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260128 15:03:14.592849 23128 log.cc:826] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222: Log is configured to *not* fsync() on all Append() calls
I20260128 15:03:14.599125 23128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260128 15:03:14.599460 23128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222: Bootstrap complete.
I20260128 15:03:14.602756 23128 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0214a26b2f4eab8246d49936b74222" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39313 } }
I20260128 15:03:14.603077 23128 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a0214a26b2f4eab8246d49936b74222, State: Initialized, Role: FOLLOWER
I20260128 15:03:14.603586 23128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0214a26b2f4eab8246d49936b74222" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39313 } }
I20260128 15:03:14.603716 23128 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:03:14.603778 23128 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:03:14.603865 23128 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 1 FOLLOWER]: Advancing to term 2
I20260128 15:03:14.605445 23128 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0214a26b2f4eab8246d49936b74222" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39313 } }
I20260128 15:03:14.605768 23128 leader_election.cc:304] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a0214a26b2f4eab8246d49936b74222; no voters:
I20260128 15:03:14.606040 23128 leader_election.cc:290] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260128 15:03:14.606148 23132 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 2 FOLLOWER]: Leader election won for term 2
I20260128 15:03:14.606335 23132 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [term 2 LEADER]: Becoming Leader. State: Replica: 8a0214a26b2f4eab8246d49936b74222, State: Running, Role: LEADER
I20260128 15:03:14.606704 23132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0214a26b2f4eab8246d49936b74222" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39313 } }
I20260128 15:03:14.607187 23128 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:03:14.607635 23132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a0214a26b2f4eab8246d49936b74222. Latest consensus state: current_term: 2 leader_uuid: "8a0214a26b2f4eab8246d49936b74222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0214a26b2f4eab8246d49936b74222" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39313 } } }
I20260128 15:03:14.607810 23132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:14.607980 23133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8a0214a26b2f4eab8246d49936b74222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a0214a26b2f4eab8246d49936b74222" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 39313 } } }
I20260128 15:03:14.608086 23133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222 [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:14.608214 23137 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:03:14.611507 23137 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:03:14.612233 23137 catalog_manager.cc:1269] Loaded cluster ID: 4188ab20ec7c4a44a3ac26b9401692c0
I20260128 15:03:14.612301 23137 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:03:14.614763 23137 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:03:14.615482 23137 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 8a0214a26b2f4eab8246d49936b74222: Loaded TSK: 0
I20260128 15:03:14.616789 23137 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260128 15:03:14.688299 23130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:14.688577 23130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:14.688659 23130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:14.691910 23130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:14.691985 23130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:14.692087 23130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:03:14.696457 23130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/oryx_127.13.88.129.keytab
--principal=oryx/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:39845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=40907
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:39313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:14.697628 23130 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:14.698479 23130 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:14.704890 23154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:14.704892 23157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:14.704890 23155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:14.705021 23130 server_base.cc:1060] running on GCE node
I20260128 15:03:14.705436 23130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:14.705988 23130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:14.707135 23130 hybrid_clock.cc:648] HybridClock initialized: now 1769612594707111 us; error 44 us; skew 500 ppm
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612594, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:14.709780 23130 init.cc:377] Logged in from keytab as oryx/127.13.88.129@KRBTEST.COM (short username oryx)
I20260128 15:03:14.710839 23130 webserver.cc:492] Webserver started at http://127.13.88.129:40907/ using document root <none> and password file <none>
I20260128 15:03:14.711387 23130 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:14.711453 23130 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:14.715102 23130 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:03:14.717006 23164 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:14.718034 23130 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:03:14.718129 23130 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "5f6e324bf3e348eeb32f0fc1d47d25c8"
format_stamp: "Formatted at 2026-01-28 15:03:13 on dist-test-slave-m28t"
server_key: "a25c00a35769f9621e95c1fda47c45e1"
server_key_iv: "680682a989a7256f60e01a07a6927025"
server_key_version: "encryptionkey@0"
I20260128 15:03:14.718554 23130 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:14.742666 23130 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:14.745635 23130 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:14.745806 23130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:14.746382 23130 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:14.747292 23130 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:14.747339 23130 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:14.747377 23130 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:14.747404 23130 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:14.757506 23130 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:39845
I20260128 15:03:14.757534 23277 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:39845 every 8 connection(s)
I20260128 15:03:14.758543 23130 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:03:14.765978 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 23130
I20260128 15:03:14.766494 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:35191
--local_ip_for_outbound_sockets=127.13.88.130
--tserver_master_addrs=127.13.88.190:39313
--webserver_port=35969
--webserver_interface=127.13.88.130
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.88.130
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/oryx_127.13.88.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
W20260128 15:03:14.770166 23280 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:14.760571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:38765 (local address 127.13.88.190:39313)
0128 15:03:14.760987 (+ 416us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:14.760999 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:14.761306 (+ 307us) server_negotiation.cc:408] Connection header received
0128 15:03:14.762341 (+ 1035us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:14.762352 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:14.762619 (+ 267us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:14.762897 (+ 278us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:14.764512 (+ 1615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.765267 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:14.766034 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.766319 (+ 285us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:14.769604 (+ 3285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:38765: BlockingRecv error: recv got EOF from 127.13.88.129:38765 (error 108)
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":113,"threads_started":1}
I20260128 15:03:14.770170 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:14.760716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:38765)
0128 15:03:14.761157 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:14.761194 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:14.761945 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:14.762990 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:14.762999 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:14.763433 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:14.764323 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:14.764338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.765404 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:14.765407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:14.765886 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:14.765896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.766126 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:14.766879 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:14.766919 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:14.769408 (+ 2489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":121,"threads_started":1}
W20260128 15:03:14.770988 23278 heartbeater.cc:646] Failed to heartbeat to 127.13.88.190:39313 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.88.190:39313: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:14.775614 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:14.771418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:33407)
0128 15:03:14.771592 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:14.771605 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:14.771688 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:14.771910 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:14.771913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:14.772057 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:14.772224 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:14.772230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.772978 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:14.772980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:14.773347 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:14.773354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.773449 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:14.773923 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:14.773936 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:14.775495 (+ 1559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101}
W20260128 15:03:14.775727 23280 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:14.771482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:33407 (local address 127.13.88.190:39313)
0128 15:03:14.771594 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:14.771598 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:14.771647 (+ 49us) server_negotiation.cc:408] Connection header received
0128 15:03:14.771774 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:14.771777 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:14.771821 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:14.771883 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:14.772333 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.772876 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:14.773444 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.773630 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:14.775626 (+ 1996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:33407: BlockingRecv error: recv got EOF from 127.13.88.129:33407 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:14.780126 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:14.776232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:59881)
0128 15:03:14.776397 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:14.776410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:14.776514 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:14.776714 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:14.776717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:14.776847 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:14.776991 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:14.776996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.777640 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:14.777643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:14.777984 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:14.777990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.778082 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:14.778567 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:14.778579 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:14.780012 (+ 1433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20260128 15:03:14.780293 23278 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260128 15:03:14.780357 23280 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:14.776296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:59881 (local address 127.13.88.190:39313)
0128 15:03:14.776398 (+ 102us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:14.776402 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:14.776485 (+ 83us) server_negotiation.cc:408] Connection header received
0128 15:03:14.776587 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:14.776590 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:14.776631 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:14.776691 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:14.777089 (+ 398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.777547 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:14.778080 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.778251 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:14.780134 (+ 1883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:59881: BlockingRecv error: recv got EOF from 127.13.88.129:59881 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260128 15:03:14.878288 23282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:14.878526 23282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:14.878595 23282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:14.882109 23282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:14.882176 23282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:14.882287 23282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:03:14.886540 23282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/oryx_127.13.88.130.keytab
--principal=oryx/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:35191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=35969
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:39313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:14.887602 23282 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:14.888484 23282 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:14.894970 23288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:14.895094 23282 server_base.cc:1060] running on GCE node
W20260128 15:03:14.894956 23287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:14.894959 23290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:14.895581 23282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:14.896100 23282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:14.897233 23282 hybrid_clock.cc:648] HybridClock initialized: now 1769612594897203 us; error 42 us; skew 500 ppm
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612594, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:14.899574 23282 init.cc:377] Logged in from keytab as oryx/127.13.88.130@KRBTEST.COM (short username oryx)
I20260128 15:03:14.900684 23282 webserver.cc:492] Webserver started at http://127.13.88.130:35969/ using document root <none> and password file <none>
I20260128 15:03:14.901368 23282 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:14.901446 23282 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:14.904929 23282 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260128 15:03:14.906674 23297 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:14.907655 23282 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:03:14.907750 23282 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "2215a7809b60430bb4d619b3baa9fdb1"
format_stamp: "Formatted at 2026-01-28 15:03:14 on dist-test-slave-m28t"
server_key: "08d4670981029d15983afe4337282002"
server_key_iv: "a438ba1e88aa87e02e3c8b36f3b5f18e"
server_key_version: "encryptionkey@0"
I20260128 15:03:14.908169 23282 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:14.929368 23282 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:14.932091 23282 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:14.932257 23282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:14.932832 23282 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:14.933714 23282 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:14.933760 23282 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:14.933799 23282 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:14.933826 23282 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:14.942348 23282 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:35191
I20260128 15:03:14.942382 23410 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:35191 every 8 connection(s)
I20260128 15:03:14.943363 23282 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
W20260128 15:03:14.952806 23280 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:14.945173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:41417 (local address 127.13.88.190:39313)
0128 15:03:14.945309 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:14.945312 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:14.946020 (+ 708us) server_negotiation.cc:408] Connection header received
0128 15:03:14.946813 (+ 793us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:14.946816 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:14.946869 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:14.946941 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:14.948632 (+ 1691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.949130 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:14.949747 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.949878 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:14.952701 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:41417: BlockingRecv error: recv got EOF from 127.13.88.130:41417 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260128 15:03:14.953122 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 23282
I20260128 15:03:14.953267 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:14.945444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:41417)
0128 15:03:14.945870 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:14.945906 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:14.946635 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:14.947084 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:14.947092 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:14.947488 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:14.948459 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:14.948477 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.949236 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:14.949238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:14.949660 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:14.949666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.949845 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:14.950418 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:14.950446 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:14.952566 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
I20260128 15:03:14.953543 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:35731
--local_ip_for_outbound_sockets=127.13.88.131
--tserver_master_addrs=127.13.88.190:39313
--webserver_port=36821
--webserver_interface=127.13.88.131
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.88.131
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/oryx_127.13.88.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:03:14.953775 23411 heartbeater.cc:646] Failed to heartbeat to 127.13.88.190:39313 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.88.190:39313: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:14.958806 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:14.954196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:43063)
0128 15:03:14.954372 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:14.954385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:14.954466 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:14.954751 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:14.954755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:14.954891 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:14.955061 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:14.955066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.955766 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:14.955769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:14.956136 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:14.956142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.956250 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:14.956860 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:14.956874 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:14.958703 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109}
W20260128 15:03:14.959007 23280 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:14.954292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:43063 (local address 127.13.88.190:39313)
0128 15:03:14.954414 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:14.954418 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:14.954432 (+ 14us) server_negotiation.cc:408] Connection header received
0128 15:03:14.954565 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:14.954569 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:14.954618 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:14.954679 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:14.955173 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.955666 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:14.956244 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.956452 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:14.958884 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:43063: BlockingRecv error: recv got EOF from 127.13.88.130:43063 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:14 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:14.964004 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:14.959614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:51149)
0128 15:03:14.959777 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:14.959791 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:14.959868 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:14.960078 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:14.960081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:14.960228 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:14.960386 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:14.960391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.961094 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:14.961098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:14.961452 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:14.961458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.961565 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:14.962100 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:14.962113 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:14.963901 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260128 15:03:14.964188 23280 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:14.959712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:51149 (local address 127.13.88.190:39313)
0128 15:03:14.959832 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:14.959837 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:14.959849 (+ 12us) server_negotiation.cc:408] Connection header received
0128 15:03:14.959945 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:14.959949 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:14.959991 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:14.960052 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:14.960516 (+ 464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.960992 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:14.961570 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:14.961747 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:14.964071 (+ 2324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:51149: BlockingRecv error: recv got EOF from 127.13.88.130:51149 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260128 15:03:14.964249 23411 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260128 15:03:15.062695 23414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:15.062937 23414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:15.062996 23414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:15.066251 23414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:15.066329 23414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:15.066411 23414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:03:15.070777 23414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/krb5kdc/oryx_127.13.88.131.keytab
--principal=oryx/127.13.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:35731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=36821
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:39313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:15.071887 23414 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:15.072763 23414 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:15.079268 23420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:15.079262 23419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:15.079465 23414 server_base.cc:1060] running on GCE node
W20260128 15:03:15.079262 23422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:15.079952 23414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:15.080547 23414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:15.081686 23414 hybrid_clock.cc:648] HybridClock initialized: now 1769612595081665 us; error 37 us; skew 500 ppm
Jan 28 15:03:15 dist-test-slave-m28t krb5kdc[22538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612595, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:15.084060 23414 init.cc:377] Logged in from keytab as oryx/127.13.88.131@KRBTEST.COM (short username oryx)
I20260128 15:03:15.085165 23414 webserver.cc:492] Webserver started at http://127.13.88.131:36821/ using document root <none> and password file <none>
I20260128 15:03:15.085713 23414 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:15.085773 23414 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:15.089272 23414 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260128 15:03:15.091132 23429 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:15.092159 23414 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:03:15.092252 23414 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "872260e365b240189919f7f8b6867bbe"
format_stamp: "Formatted at 2026-01-28 15:03:14 on dist-test-slave-m28t"
server_key: "c070bfe141ca550a5aa047be32cf5d40"
server_key_iv: "6eff377076f73b809e283db301de1b53"
server_key_version: "encryptionkey@0"
I20260128 15:03:15.092710 23414 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:15.105371 23414 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:15.108297 23414 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:15.108515 23414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:15.109118 23414 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:15.110076 23414 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:15.110123 23414 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:15.110163 23414 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:15.110183 23414 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:15.119038 23414 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:35731
I20260128 15:03:15.119055 23542 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:35731 every 8 connection(s)
I20260128 15:03:15.120107 23414 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
Jan 28 15:03:15 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:15 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:15.129428 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 23414
W20260128 15:03:15.130025 23280 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:15.121984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:44895 (local address 127.13.88.190:39313)
0128 15:03:15.122127 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:15.122131 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:15.122870 (+ 739us) server_negotiation.cc:408] Connection header received
0128 15:03:15.123722 (+ 852us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:15.123725 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:15.123775 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:15.123850 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:15.125532 (+ 1682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.126019 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:15.126607 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.126732 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:15.129900 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:44895: BlockingRecv error: recv got EOF from 127.13.88.131:44895 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260128 15:03:15.130652 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:15.122266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:44895)
0128 15:03:15.122724 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:15.122771 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:15.123523 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:15.124001 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:15.124009 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:15.124405 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:15.125378 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:15.125392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.126105 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:15.126108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:15.126516 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:15.126523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.126700 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:15.127194 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:15.127216 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:15.129761 (+ 2545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":103,"threads_started":1}
W20260128 15:03:15.131080 23543 heartbeater.cc:646] Failed to heartbeat to 127.13.88.190:39313 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.88.190:39313: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Jan 28 15:03:15 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:15 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:15.136641 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:15.131469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:38395)
0128 15:03:15.132201 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:15.132212 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:15.132283 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:15.132543 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:15.132546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:15.132686 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:15.132891 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:15.132899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.133677 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:15.133681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:15.134172 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:15.134180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.134318 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:15.134756 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:15.134768 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:15.136545 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260128 15:03:15.136817 23547 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:15.131536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:38395 (local address 127.13.88.190:39313)
0128 15:03:15.131761 (+ 225us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:15.131764 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:15.132269 (+ 505us) server_negotiation.cc:408] Connection header received
0128 15:03:15.132371 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:15.132374 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:15.132443 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:15.132522 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:15.133010 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.133569 (+ 559us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:15.134273 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.134456 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:15.136688 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:38395: BlockingRecv error: recv got EOF from 127.13.88.131:38395 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
I20260128 15:03:15.138298 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:15.130299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48676 (local address 127.13.88.190:39313)
0128 15:03:15.130405 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:15.130409 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:15.130424 (+ 15us) server_negotiation.cc:408] Connection header received
0128 15:03:15.130521 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:15.130531 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:15.130581 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:15.130651 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:15.131592 (+ 941us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.132088 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:15.132793 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.132968 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:15.134034 (+ 1066us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:15.134050 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:15.134062 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:15.134103 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:15.135962 (+ 1859us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:15.136470 (+ 508us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:15.136477 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:15.136485 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:15.136596 (+ 111us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:15.137121 (+ 525us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:15.137124 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:15.137131 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:15.137500 (+ 369us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:15.137707 (+ 207us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:15.137927 (+ 220us) server_negotiation.cc:300] Negotiation successful
0128 15:03:15.138184 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Jan 28 15:03:15 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:15 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:15.141760 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:15.137256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:58359)
0128 15:03:15.137419 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:15.137432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:15.137492 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:15.137711 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:15.137714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:15.137859 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:15.138052 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:15.138060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.138856 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:15.138859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:15.139257 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:15.139266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.139373 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:15.139854 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:15.139869 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:15.141664 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260128 15:03:15.141875 23547 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:15.137357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:58359 (local address 127.13.88.190:39313)
0128 15:03:15.137497 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:15.137501 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:15.137538 (+ 37us) server_negotiation.cc:408] Connection header received
0128 15:03:15.137570 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:15.137573 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:15.137613 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:15.137673 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:15.138168 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.138751 (+ 583us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:15.139416 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.139602 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:15.141782 (+ 2180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:58359: BlockingRecv error: recv got EOF from 127.13.88.131:58359 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260128 15:03:15.142005 23543 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jan 28 15:03:15 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:15 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:15.785732 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:15.780938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:46901)
0128 15:03:15.781219 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:15.781233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:15.781337 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:15.781561 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:15.781563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:15.781773 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:15.781973 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:15.781980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.782759 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:15.782761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:15.783138 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:15.783143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.783225 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:15.783753 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:15.783766 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:15.785606 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":84,"threads_started":1}
W20260128 15:03:15.785866 23549 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:15.781026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:46901 (local address 127.13.88.190:39313)
0128 15:03:15.781254 (+ 228us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:15.781257 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:15.781307 (+ 50us) server_negotiation.cc:408] Connection header received
0128 15:03:15.781416 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:15.781418 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:15.781466 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:15.781535 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:15.782117 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.782671 (+ 554us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:15.783250 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.783385 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:15.785733 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:46901: BlockingRecv error: recv got EOF from 127.13.88.129:46901 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
Jan 28 15:03:15 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:15 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:15.969544 23550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:15.964861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:57515)
0128 15:03:15.965144 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:15.965157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:15.965258 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:15.965512 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:15.965515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:15.965751 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:15.965981 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:15.965989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.966731 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:15.966734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:15.967130 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:15.967136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.967222 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:15.967715 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:15.967728 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:15.969422 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":88,"threads_started":1}
W20260128 15:03:15.969682 23549 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:15.964999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:57515 (local address 127.13.88.190:39313)
0128 15:03:15.965128 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:15.965132 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:15.965223 (+ 91us) server_negotiation.cc:408] Connection header received
0128 15:03:15.965329 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:15.965332 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:15.965392 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:15.965487 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:15.966096 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.966635 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:15.967232 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:15.967377 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:15.969565 (+ 2188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:57515: BlockingRecv error: recv got EOF from 127.13.88.130:57515 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jan 28 15:03:16 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:16 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:16.147300 23551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:16.142649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:40675)
0128 15:03:16.142937 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:16.142949 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:16.143062 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:16.143304 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:16.143307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:16.143524 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:16.143731 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:16.143738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:16.144503 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:16.144507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:16.144902 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:16.144910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:16.145010 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:16.145513 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:16.145526 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:16.147150 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":101,"threads_started":1}
W20260128 15:03:16.147409 23549 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:16.142729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:40675 (local address 127.13.88.190:39313)
0128 15:03:16.142866 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:16.142871 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:16.143022 (+ 151us) server_negotiation.cc:408] Connection header received
0128 15:03:16.143151 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:16.143154 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:16.143203 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:16.143267 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:16.143872 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:16.144387 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:16.145009 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:16.145188 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:16.147283 (+ 2095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:40675: BlockingRecv error: recv got EOF from 127.13.88.131:40675 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jan 28 15:03:16 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:16 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:16.796381 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:16.786583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:56581)
0128 15:03:16.786833 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:16.786846 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:16.786940 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:16.787149 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:16.787151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:16.787340 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:16.787539 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:16.787545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:16.788292 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:16.788294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:16.788749 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:16.788757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:16.788846 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:16.794549 (+ 5703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:16.794575 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:16.796259 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":84,"threads_started":1}
W20260128 15:03:16.796620 23553 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:16.786629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:56581 (local address 127.13.88.190:39313)
0128 15:03:16.786858 (+ 229us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:16.786862 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:16.786895 (+ 33us) server_negotiation.cc:408] Connection header received
0128 15:03:16.787009 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:16.787011 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:16.787055 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:16.787130 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:16.787655 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:16.788178 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:16.788853 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:16.788999 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:16.796488 (+ 7489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:56581: BlockingRecv error: recv got EOF from 127.13.88.129:56581 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
Jan 28 15:03:16 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:16 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:16.978793 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:16.970236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:36001)
0128 15:03:16.970498 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:16.970513 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:16.970629 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:16.970857 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:16.970860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:16.971050 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:16.971235 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:16.971241 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:16.971952 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:16.971955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:16.972320 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:16.972326 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:16.972455 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:16.976960 (+ 4505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:16.976974 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:16.978680 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
W20260128 15:03:16.978924 23553 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:16.970316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:36001 (local address 127.13.88.190:39313)
0128 15:03:16.970457 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:16.970461 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:16.970577 (+ 116us) server_negotiation.cc:408] Connection header received
0128 15:03:16.970711 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:16.970714 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:16.970763 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:16.970829 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:16.971363 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:16.971854 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:16.972448 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:16.972625 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:16.978808 (+ 6183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:36001: BlockingRecv error: recv got EOF from 127.13.88.130:36001 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jan 28 15:03:17 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:17 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:17.152976 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:17.148096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:41701)
0128 15:03:17.148408 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:17.148445 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:17.148573 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:17.148836 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:17.148839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:17.149029 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:17.149220 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:17.149226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:17.149980 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:17.149984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:17.150363 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:17.150371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:17.150490 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:17.151027 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:17.151042 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:17.152850 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":76,"threads_started":1}
W20260128 15:03:17.153097 23553 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:17.148299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:41701 (local address 127.13.88.190:39313)
0128 15:03:17.148505 (+ 206us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:17.148509 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:17.148524 (+ 15us) server_negotiation.cc:408] Connection header received
0128 15:03:17.148675 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:17.148678 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:17.148724 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:17.148792 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:17.149349 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:17.149865 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:17.150487 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:17.150664 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:17.152993 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:41701: BlockingRecv error: recv got EOF from 127.13.88.131:41701 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Jan 28 15:03:17 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:17 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:17.801846 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:17.797124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:60287)
0128 15:03:17.797425 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:17.797438 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:17.797535 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:17.797746 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:17.797748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:17.797923 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:17.798125 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:17.798131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:17.798916 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:17.798919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:17.799301 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:17.799307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:17.799401 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:17.799909 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:17.799921 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:17.801696 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":76,"threads_started":1}
W20260128 15:03:17.801957 23557 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:17.797192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:60287 (local address 127.13.88.190:39313)
0128 15:03:17.797451 (+ 259us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:17.797454 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:17.797484 (+ 30us) server_negotiation.cc:408] Connection header received
0128 15:03:17.797601 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:17.797604 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:17.797649 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:17.797727 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:17.798267 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:17.798807 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:17.799427 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:17.799579 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:17.801846 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:60287: BlockingRecv error: recv got EOF from 127.13.88.129:60287 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":68,"threads_started":1}
Jan 28 15:03:17 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:17 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:17.984330 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:17.979542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:57601)
0128 15:03:17.979805 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:17.979819 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:17.979933 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:17.980155 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:17.980157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:17.980329 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:17.980600 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:17.980607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:17.981505 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:17.981508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:17.982030 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:17.982040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:17.982134 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:17.982629 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:17.982642 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:17.984219 (+ 1577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
W20260128 15:03:17.984483 23557 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:17.979633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:57601 (local address 127.13.88.190:39313)
0128 15:03:17.979807 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:17.979811 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:17.979893 (+ 82us) server_negotiation.cc:408] Connection header received
0128 15:03:17.980014 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:17.980017 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:17.980066 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:17.980131 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:17.980747 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:17.981392 (+ 645us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:17.982136 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:17.982278 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:17.984344 (+ 2066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:57601: BlockingRecv error: recv got EOF from 127.13.88.130:57601 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jan 28 15:03:18 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:18 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:18.158490 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:18.153867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:36519)
0128 15:03:18.154139 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:18.154154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:18.154274 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:18.154529 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:18.154532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:18.154720 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:18.154907 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:18.154913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:18.155632 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:18.155635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:18.156013 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:18.156020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:18.156118 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:18.156680 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:18.156693 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:18.158308 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
W20260128 15:03:18.158542 23557 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:18.153980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:36519 (local address 127.13.88.190:39313)
0128 15:03:18.154125 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:18.154130 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:18.154230 (+ 100us) server_negotiation.cc:408] Connection header received
0128 15:03:18.154371 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:18.154375 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:18.154426 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:18.154488 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:18.155023 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:18.155524 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:18.156113 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:18.156303 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:18.158445 (+ 2142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:36519: BlockingRecv error: recv got EOF from 127.13.88.131:36519 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jan 28 15:03:18 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:18 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:18.807559 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:18.802739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:40037)
0128 15:03:18.803025 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:18.803038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:18.803131 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:18.803367 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:18.803370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:18.803583 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:18.803787 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:18.803794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:18.804706 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:18.804710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:18.805099 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:18.805105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:18.805199 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:18.805714 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:18.805726 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:18.807442 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":105,"threads_started":1}
W20260128 15:03:18.807708 23561 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:18.802854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:40037 (local address 127.13.88.190:39313)
0128 15:03:18.803092 (+ 238us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:18.803095 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:18.803108 (+ 13us) server_negotiation.cc:408] Connection header received
0128 15:03:18.803206 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:18.803208 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:18.803254 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:18.803328 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:18.803876 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:18.804530 (+ 654us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:18.805198 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:18.805384 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:18.807575 (+ 2191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:40037: BlockingRecv error: recv got EOF from 127.13.88.129:40037 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
Jan 28 15:03:18 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:18 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:18.989743 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:18.985128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:40513)
0128 15:03:18.985411 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:18.985424 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:18.985524 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:18.985749 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:18.985755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:18.985927 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:18.986117 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:18.986123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:18.986794 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:18.986796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:18.987166 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:18.987173 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:18.987276 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:18.987805 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:18.987822 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:18.989632 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
W20260128 15:03:18.989866 23561 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:18.985276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:40513 (local address 127.13.88.190:39313)
0128 15:03:18.985440 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:18.985444 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:18.985474 (+ 30us) server_negotiation.cc:408] Connection header received
0128 15:03:18.985602 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:18.985606 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:18.985653 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:18.985719 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:18.986205 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:18.986714 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:18.987271 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:18.987450 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:18.989754 (+ 2304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:40513: BlockingRecv error: recv got EOF from 127.13.88.130:40513 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jan 28 15:03:19 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:19 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:19.164063 23563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:19.159315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:37885)
0128 15:03:19.159574 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:19.159587 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:19.159698 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:19.159921 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:19.159924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:19.160126 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:19.160366 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:19.160374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:19.161273 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:19.161277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:19.161687 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:19.161695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:19.161794 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:19.162271 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:19.162284 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:19.163925 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
W20260128 15:03:19.164155 23561 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:19.159475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:37885 (local address 127.13.88.190:39313)
0128 15:03:19.159599 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:19.159603 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:19.159659 (+ 56us) server_negotiation.cc:408] Connection header received
0128 15:03:19.159772 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:19.159775 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:19.159825 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:19.159899 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:19.160524 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:19.161158 (+ 634us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:19.161794 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:19.161964 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:19.164056 (+ 2092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:37885: BlockingRecv error: recv got EOF from 127.13.88.131:37885 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jan 28 15:03:19 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:19 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:19.813472 23565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:19.808402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:57999)
0128 15:03:19.808888 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:19.808901 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:19.809006 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:19.809262 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:19.809264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:19.809473 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:19.809670 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:19.809676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:19.810462 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:19.810466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:19.810860 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:19.810868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:19.810969 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:19.811508 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:19.811521 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:19.813324 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":68,"threads_started":1}
W20260128 15:03:19.813694 23564 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:19.808507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:57999 (local address 127.13.88.190:39313)
0128 15:03:19.808797 (+ 290us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:19.808801 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:19.809000 (+ 199us) server_negotiation.cc:408] Connection header received
0128 15:03:19.809076 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:19.809079 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:19.809129 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:19.809245 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:19.809805 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:19.810336 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:19.810980 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:19.811126 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:19.813464 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:57999: BlockingRecv error: recv got EOF from 127.13.88.129:57999 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":82,"threads_started":1}
Jan 28 15:03:19 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:19 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:19.995105 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:19.990538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:35383)
0128 15:03:19.990831 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:19.990847 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:19.990959 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:19.991186 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:19.991189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:19.991370 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:19.991566 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:19.991572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:19.992266 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:19.992269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:19.992697 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:19.992704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:19.992790 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:19.993269 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:19.993282 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:19.994984 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":109,"threads_started":1}
W20260128 15:03:19.995257 23564 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:19.990668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:35383 (local address 127.13.88.190:39313)
0128 15:03:19.990807 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:19.990811 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:19.990922 (+ 111us) server_negotiation.cc:408] Connection header received
0128 15:03:19.991039 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:19.991042 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:19.991093 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:19.991159 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:19.991677 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:19.992181 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:19.992772 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:19.992908 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:19.995142 (+ 2234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:35383: BlockingRecv error: recv got EOF from 127.13.88.130:35383 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jan 28 15:03:20 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:20 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:20.169631 23567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:20.165016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:54339)
0128 15:03:20.165273 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:20.165286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:20.165400 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:20.165634 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:20.165636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:20.165806 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:20.165997 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:20.166003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:20.166828 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:20.166831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:20.167269 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:20.167279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:20.167386 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:20.167855 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:20.167866 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:20.169515 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
W20260128 15:03:20.169732 23564 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:20.165140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:54339 (local address 127.13.88.190:39313)
0128 15:03:20.165291 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:20.165295 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:20.165350 (+ 55us) server_negotiation.cc:408] Connection header received
0128 15:03:20.165472 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:20.165475 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:20.165526 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:20.165603 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:20.166130 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:20.166721 (+ 591us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:20.167375 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:20.167522 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:20.169635 (+ 2113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:54339: BlockingRecv error: recv got EOF from 127.13.88.131:54339 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jan 28 15:03:20 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:20 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:20.818738 23568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:20.814176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:59643)
0128 15:03:20.814457 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:20.814470 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:20.814574 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:20.814802 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:20.814804 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:20.814977 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:20.815169 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:20.815175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:20.815885 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:20.815887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:20.816256 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:20.816262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:20.816359 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:20.816884 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:20.816897 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:20.818615 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
W20260128 15:03:20.818929 23569 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:20.814242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:59643 (local address 127.13.88.190:39313)
0128 15:03:20.814483 (+ 241us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:20.814487 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:20.814536 (+ 49us) server_negotiation.cc:408] Connection header received
0128 15:03:20.814654 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:20.814657 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:20.814705 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:20.814779 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:20.815259 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:20.815802 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:20.816347 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:20.816529 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:20.818796 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:59643: BlockingRecv error: recv got EOF from 127.13.88.129:59643 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":58,"threads_started":1}
Jan 28 15:03:20 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:20 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:21.000759 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:20.995991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:33009)
0128 15:03:20.996308 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:20.996322 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:20.996478 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:20.996749 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:20.996752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:20.996939 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:20.997130 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:20.997136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:20.997907 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:20.997911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:20.998289 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:20.998295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:20.998404 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:20.998898 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:20.998912 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:21.000618 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":111,"threads_started":1}
W20260128 15:03:21.000851 23569 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:20.996059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:33009 (local address 127.13.88.190:39313)
0128 15:03:20.996193 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:20.996197 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:20.996411 (+ 214us) server_negotiation.cc:408] Connection header received
0128 15:03:20.996564 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:20.996567 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:20.996631 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:20.996703 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:20.997245 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:20.997806 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:20.998396 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:20.998572 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:21.000753 (+ 2181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:33009: BlockingRecv error: recv got EOF from 127.13.88.130:33009 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jan 28 15:03:21 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:21 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:21.174917 23571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:21.170380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:34345)
0128 15:03:21.170641 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:21.170654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:21.170761 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:21.171064 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:21.171067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:21.171239 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:21.171439 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:21.171445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:21.172113 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:21.172116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:21.172567 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:21.172575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:21.172666 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:21.173128 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:21.173141 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:21.174803 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":83,"threads_started":1}
W20260128 15:03:21.175092 23569 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:21.170431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:34345 (local address 127.13.88.190:39313)
0128 15:03:21.170579 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:21.170583 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:21.170732 (+ 149us) server_negotiation.cc:408] Connection header received
0128 15:03:21.170868 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:21.170871 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:21.170926 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:21.170994 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:21.171534 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:21.172021 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:21.172653 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:21.172772 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:21.174967 (+ 2195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:34345: BlockingRecv error: recv got EOF from 127.13.88.131:34345 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jan 28 15:03:21 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:21 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:21.828327 23572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:21.819681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:58131)
0128 15:03:21.819969 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:21.819982 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:21.820081 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:21.820348 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:21.820351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:21.820594 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:21.820847 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:21.820857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:21.821872 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:21.821876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:21.822333 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:21.822344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:21.822450 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:21.826276 (+ 3826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:21.826296 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:21.828192 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":73,"threads_started":1}
W20260128 15:03:21.828595 23573 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:21.819777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:58131 (local address 127.13.88.190:39313)
0128 15:03:21.820033 (+ 256us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:21.820037 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:21.820052 (+ 15us) server_negotiation.cc:408] Connection header received
0128 15:03:21.820162 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:21.820165 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:21.820210 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:21.820297 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:21.820970 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:21.821645 (+ 675us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:21.822450 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:21.822622 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:21.828383 (+ 5761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:58131: BlockingRecv error: recv got EOF from 127.13.88.129:58131 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
Jan 28 15:03:22 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:22 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:22.006459 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:22.001548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:53033)
0128 15:03:22.001872 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:22.001885 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:22.002007 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:22.002268 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:22.002271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:22.002489 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:22.002700 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:22.002707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:22.003443 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:22.003447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:22.003838 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:22.003845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:22.003952 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:22.004513 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:22.004526 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:22.006305 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":98,"threads_started":1}
W20260128 15:03:22.006593 23573 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:22.001636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:53033 (local address 127.13.88.190:39313)
0128 15:03:22.001784 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:22.001789 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:22.001958 (+ 169us) server_negotiation.cc:408] Connection header received
0128 15:03:22.002102 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:22.002106 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:22.002160 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:22.002229 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:22.002831 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:22.003338 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:22.003950 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:22.004137 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:22.006489 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:53033: BlockingRecv error: recv got EOF from 127.13.88.130:53033 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jan 28 15:03:22 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:22 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:22.180310 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:22.175745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:39599)
0128 15:03:22.176047 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:22.176060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:22.176162 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:22.176441 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:22.176444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:22.176623 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:22.176819 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:22.176825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:22.177510 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:22.177513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:22.177889 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:22.177895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:22.177979 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:22.178496 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:22.178509 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:22.180193 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":84,"threads_started":1}
W20260128 15:03:22.180528 23573 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:22.175817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:39599 (local address 127.13.88.190:39313)
0128 15:03:22.175958 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:22.175962 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:22.176124 (+ 162us) server_negotiation.cc:408] Connection header received
0128 15:03:22.176235 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:22.176237 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:22.176286 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:22.176389 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:22.176911 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:22.177418 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:22.177988 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:22.178132 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:22.180361 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:39599: BlockingRecv error: recv got EOF from 127.13.88.131:39599 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jan 28 15:03:22 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:22 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:22.833736 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:22.829175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:42777)
0128 15:03:22.829451 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:22.829466 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:22.829559 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:22.829822 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:22.829825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:22.829997 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:22.830193 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:22.830200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:22.830912 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:22.830915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:22.831287 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:22.831294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:22.831391 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:22.831934 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:22.831947 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:22.833591 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
W20260128 15:03:22.833848 23577 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:22.829314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:42777 (local address 127.13.88.190:39313)
0128 15:03:22.829564 (+ 250us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:22.829568 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:22.829582 (+ 14us) server_negotiation.cc:408] Connection header received
0128 15:03:22.829655 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:22.829658 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:22.829704 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:22.829781 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:22.830285 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:22.830800 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:22.831428 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:22.831613 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:22.833741 (+ 2128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:42777: BlockingRecv error: recv got EOF from 127.13.88.129:42777 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":95,"threads_started":1}
Jan 28 15:03:23 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:23 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:23.012461 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:23.007420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:34269)
0128 15:03:23.007718 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:23.007732 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:23.007852 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:23.008087 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:23.008090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:23.008320 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:23.008588 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:23.008596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:23.009561 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:23.009564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:23.009949 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:23.009956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:23.010048 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:23.010542 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:23.010556 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:23.012291 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":98,"threads_started":1}
W20260128 15:03:23.012598 23577 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:23.007597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:34269 (local address 127.13.88.190:39313)
0128 15:03:23.007751 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:23.007755 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:23.007812 (+ 57us) server_negotiation.cc:408] Connection header received
0128 15:03:23.007938 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:23.007942 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:23.007990 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:23.008055 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:23.008694 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:23.009468 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:23.010035 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:23.010166 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:23.012469 (+ 2303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:34269: BlockingRecv error: recv got EOF from 127.13.88.130:34269 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jan 28 15:03:23 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:23 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:23.186096 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:23.181162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:37937)
0128 15:03:23.181520 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:23.181536 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:23.181663 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:23.181906 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:23.181909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:23.182096 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:23.182286 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:23.182292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:23.183014 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:23.183018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:23.183411 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:23.183418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:23.183517 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:23.184083 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:23.184097 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:23.185953 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":112,"threads_started":1}
W20260128 15:03:23.186170 23577 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:23.181323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:37937 (local address 127.13.88.190:39313)
0128 15:03:23.181520 (+ 197us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:23.181524 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:23.181616 (+ 92us) server_negotiation.cc:408] Connection header received
0128 15:03:23.181750 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:23.181754 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:23.181802 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:23.181868 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:23.182417 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:23.182911 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:23.183518 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:23.183704 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:23.186077 (+ 2373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:37937: BlockingRecv error: recv got EOF from 127.13.88.131:37937 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jan 28 15:03:23 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:23 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:23.839548 23580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:23.834505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:37117)
0128 15:03:23.834833 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:23.834848 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:23.834963 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:23.835211 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:23.835214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:23.835429 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:23.835632 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:23.835640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:23.836400 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:23.836404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:23.836819 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:23.836826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:23.836936 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:23.837533 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:23.837547 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:23.839427 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":90,"threads_started":1}
W20260128 15:03:23.839769 23581 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:23.834572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:37117 (local address 127.13.88.190:39313)
0128 15:03:23.834834 (+ 262us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:23.834839 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:23.834905 (+ 66us) server_negotiation.cc:408] Connection header received
0128 15:03:23.835049 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:23.835053 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:23.835103 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:23.835175 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:23.835760 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:23.836274 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:23.836932 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:23.837115 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:23.839625 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:37117: BlockingRecv error: recv got EOF from 127.13.88.129:37117 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":64,"threads_started":1}
Jan 28 15:03:24 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:24 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:24.017951 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:24.013152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:45701)
0128 15:03:24.013475 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:24.013491 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:24.013618 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:24.013836 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:24.013838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:24.014009 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:24.014200 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:24.014207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:24.014945 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:24.014948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:24.015324 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:24.015331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:24.015442 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:24.015994 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:24.016007 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:24.017835 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":81,"threads_started":1}
W20260128 15:03:24.018177 23581 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:24.013236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:45701 (local address 127.13.88.190:39313)
0128 15:03:24.013401 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:24.013406 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:24.013564 (+ 158us) server_negotiation.cc:408] Connection header received
0128 15:03:24.013693 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:24.013696 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:24.013746 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:24.013811 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:24.014303 (+ 492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:24.014838 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:24.015448 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:24.015596 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:24.017971 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:45701: BlockingRecv error: recv got EOF from 127.13.88.130:45701 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jan 28 15:03:24 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:24 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:24.191859 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:24.186987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:37177)
0128 15:03:24.187273 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:24.187287 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:24.187407 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:24.187689 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:24.187693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:24.187883 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:24.188077 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:24.188084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:24.188862 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:24.188865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:24.189258 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:24.189264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:24.189364 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:24.189954 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:24.189974 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:24.191709 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
W20260128 15:03:24.191964 23581 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:24.187115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:37177 (local address 127.13.88.190:39313)
0128 15:03:24.187274 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:24.187279 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:24.187382 (+ 103us) server_negotiation.cc:408] Connection header received
0128 15:03:24.187521 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:24.187524 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:24.187573 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:24.187652 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:24.188202 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:24.188755 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:24.189370 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:24.189548 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:24.191862 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:37177: BlockingRecv error: recv got EOF from 127.13.88.131:37177 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jan 28 15:03:24 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:24 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:24.844902 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:24.840375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:41513)
0128 15:03:24.840656 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:24.840668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:24.840757 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:24.841000 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:24.841002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:24.841171 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:24.841362 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:24.841368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:24.842085 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:24.842088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:24.842470 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:24.842476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:24.842557 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:24.843007 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:24.843019 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:24.844768 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":104,"threads_started":1}
W20260128 15:03:24.845027 23585 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:24.840488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:41513 (local address 127.13.88.190:39313)
0128 15:03:24.840710 (+ 222us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:24.840714 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:24.840729 (+ 15us) server_negotiation.cc:408] Connection header received
0128 15:03:24.840842 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:24.840845 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:24.840889 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:24.840970 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:24.841463 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:24.841998 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:24.842545 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:24.842680 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:24.844909 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:41513: BlockingRecv error: recv got EOF from 127.13.88.129:41513 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":67,"threads_started":1}
Jan 28 15:03:25 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:25 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:25.023512 23586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:25.018736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:43021)
0128 15:03:25.019007 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:25.019020 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:25.019130 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:25.019361 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:25.019364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:25.019554 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:25.019755 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:25.019761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:25.020707 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:25.020711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:25.021147 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:25.021155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:25.021275 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:25.021766 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:25.021780 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:25.023392 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
W20260128 15:03:25.023617 23585 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:25.018817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:43021 (local address 127.13.88.190:39313)
0128 15:03:25.018973 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:25.018977 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:25.019088 (+ 111us) server_negotiation.cc:408] Connection header received
0128 15:03:25.019214 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:25.019217 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:25.019266 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:25.019329 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:25.019870 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:25.020588 (+ 718us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:25.021254 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:25.021425 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:25.023511 (+ 2086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:43021: BlockingRecv error: recv got EOF from 127.13.88.130:43021 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jan 28 15:03:25 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:25 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:25.197691 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:25.192780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:42649)
0128 15:03:25.193051 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:25.193063 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:25.193148 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:25.193431 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:25.193434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:25.193644 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:25.193852 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:25.193859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:25.194698 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:25.194701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:25.195076 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:25.195083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:25.195180 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:25.195686 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:25.195698 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:25.197536 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
W20260128 15:03:25.197762 23585 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:25.192947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:42649 (local address 127.13.88.190:39313)
0128 15:03:25.193083 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:25.193087 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:25.193107 (+ 20us) server_negotiation.cc:408] Connection header received
0128 15:03:25.193240 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:25.193244 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:25.193300 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:25.193393 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:25.193962 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:25.194579 (+ 617us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:25.195200 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:25.195342 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:25.197668 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:42649: BlockingRecv error: recv got EOF from 127.13.88.131:42649 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jan 28 15:03:25 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:25 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:25.850874 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:25.845801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:39163)
0128 15:03:25.846114 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:25.846128 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:25.846236 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:25.846525 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:25.846528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:25.846730 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:25.846927 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:25.846933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:25.847676 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:25.847679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:25.848079 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:25.848087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:25.848182 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:25.848761 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:25.848775 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:25.850760 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":107,"threads_started":1}
W20260128 15:03:25.851028 23589 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:25.845943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:39163 (local address 127.13.88.190:39313)
0128 15:03:25.846182 (+ 239us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:25.846186 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:25.846201 (+ 15us) server_negotiation.cc:408] Connection header received
0128 15:03:25.846320 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:25.846323 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:25.846369 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:25.846448 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:25.847030 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:25.847543 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:25.848194 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:25.848334 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:25.850901 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:39163: BlockingRecv error: recv got EOF from 127.13.88.129:39163 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":65,"threads_started":1}
Jan 28 15:03:26 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:26 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:26.029218 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:26.024288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:42959)
0128 15:03:26.024624 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:26.024638 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:26.024765 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:26.025018 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:26.025021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:26.025230 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:26.025471 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:26.025481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:26.026198 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:26.026202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:26.026605 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:26.026612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:26.026715 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:26.027258 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:26.027272 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:26.029101 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":143,"threads_started":1}
W20260128 15:03:26.029366 23589 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:26.024452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:42959 (local address 127.13.88.190:39313)
0128 15:03:26.024598 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:26.024604 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:26.024730 (+ 126us) server_negotiation.cc:408] Connection header received
0128 15:03:26.024851 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:26.024854 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:26.024904 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:26.024975 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:26.025589 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:26.026095 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:26.026712 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:26.026902 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:26.029251 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:42959: BlockingRecv error: recv got EOF from 127.13.88.130:42959 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jan 28 15:03:26 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:26 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:26.203277 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:26.198509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:33573)
0128 15:03:26.198790 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:26.198803 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:26.198912 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:26.199144 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:26.199146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:26.199316 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:26.199528 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:26.199534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:26.200252 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:26.200254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:26.200700 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:26.200710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:26.200815 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:26.201359 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:26.201372 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:26.203161 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
W20260128 15:03:26.203384 23589 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:26.198604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:33573 (local address 127.13.88.190:39313)
0128 15:03:26.198738 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:26.198741 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:26.198869 (+ 128us) server_negotiation.cc:408] Connection header received
0128 15:03:26.198987 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:26.198990 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:26.199039 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:26.199118 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:26.199644 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:26.200151 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:26.200810 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:26.200976 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:26.203284 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:33573: BlockingRecv error: recv got EOF from 127.13.88.131:33573 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jan 28 15:03:26 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:26 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:26.861431 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:26.851634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:54217)
0128 15:03:26.851948 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:26.851962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:26.852069 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:26.852331 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:26.852333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:26.852541 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:26.852752 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:26.852759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:26.853646 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:26.853650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:26.854169 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:26.854180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:26.854294 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:26.859571 (+ 5277us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:26.859587 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:26.861299 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":92,"threads_started":1}
W20260128 15:03:26.861589 23593 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:26.851748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:54217 (local address 127.13.88.190:39313)
0128 15:03:26.851994 (+ 246us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:26.851997 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:26.852017 (+ 20us) server_negotiation.cc:408] Connection header received
0128 15:03:26.852161 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:26.852163 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:26.852212 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:26.852292 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:26.852893 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:26.853506 (+ 613us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:26.854290 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:26.854479 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:26.861460 (+ 6981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:54217: BlockingRecv error: recv got EOF from 127.13.88.129:54217 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":60,"threads_started":1}
Jan 28 15:03:27 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:27 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:27.034904 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:27.030054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:34961)
0128 15:03:27.030312 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:27.030325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:27.030458 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:27.030711 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:27.030714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:27.030904 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:27.031093 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:27.031099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:27.031839 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:27.031842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:27.032230 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:27.032237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:27.032342 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:27.032952 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:27.032967 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:27.034752 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":93,"threads_started":1}
W20260128 15:03:27.035125 23593 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:27.030106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:34961 (local address 127.13.88.190:39313)
0128 15:03:27.030253 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:27.030258 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:27.030410 (+ 152us) server_negotiation.cc:408] Connection header received
0128 15:03:27.030544 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:27.030547 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:27.030600 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:27.030671 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:27.031214 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:27.031730 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:27.032339 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:27.032562 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:27.034906 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:34961: BlockingRecv error: recv got EOF from 127.13.88.130:34961 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jan 28 15:03:27 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:27 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:27.208966 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:27.204100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:48701)
0128 15:03:27.204363 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:27.204376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:27.204518 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:27.204778 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:27.204780 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:27.204957 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:27.205158 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:27.205164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:27.205877 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:27.205879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:27.206260 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:27.206266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:27.206356 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:27.206886 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:27.206898 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:27.208819 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
W20260128 15:03:27.209115 23593 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:27.204222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:48701 (local address 127.13.88.190:39313)
0128 15:03:27.204393 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:27.204396 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:27.204458 (+ 62us) server_negotiation.cc:408] Connection header received
0128 15:03:27.204619 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:27.204622 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:27.204673 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:27.204750 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:27.205264 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:27.205784 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:27.206358 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:27.206511 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:27.208998 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:48701: BlockingRecv error: recv got EOF from 127.13.88.131:48701 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jan 28 15:03:27 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:27 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:27.866781 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:27.862266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:39867)
0128 15:03:27.862541 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:27.862554 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:27.862656 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:27.862870 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:27.862872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:27.863046 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:27.863244 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:27.863250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:27.864006 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:27.864009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:27.864397 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:27.864404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:27.864540 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:27.865017 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:27.865029 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:27.866665 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
W20260128 15:03:27.866961 23597 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:27.862306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:39867 (local address 127.13.88.190:39313)
0128 15:03:27.862540 (+ 234us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:27.862543 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:27.862605 (+ 62us) server_negotiation.cc:408] Connection header received
0128 15:03:27.862726 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:27.862728 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:27.862773 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:27.862845 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:27.863345 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:27.863894 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:27.864538 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:27.864679 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:27.866833 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:39867: BlockingRecv error: recv got EOF from 127.13.88.129:39867 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":73,"threads_started":1}
Jan 28 15:03:28 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:28 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:28.040650 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:28.035760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:59323)
0128 15:03:28.036053 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:28.036069 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:28.036192 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:28.036491 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:28.036494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:28.036695 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:28.036887 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:28.036893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:28.037628 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:28.037632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:28.038006 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:28.038013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:28.038119 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:28.038756 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:28.038770 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:28.040537 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":85,"threads_started":1}
W20260128 15:03:28.040871 23597 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:28.035954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:59323 (local address 127.13.88.190:39313)
0128 15:03:28.036093 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:28.036098 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:28.036132 (+ 34us) server_negotiation.cc:408] Connection header received
0128 15:03:28.036306 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:28.036308 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:28.036356 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:28.036466 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:28.037017 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:28.037518 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:28.038116 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:28.038310 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:28.040663 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:59323: BlockingRecv error: recv got EOF from 127.13.88.130:59323 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jan 28 15:03:28 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:28 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:28.214387 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:28.209761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:44125)
0128 15:03:28.210014 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:28.210028 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:28.210130 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:28.210330 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:28.210332 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:28.210543 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:28.210791 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:28.210799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:28.211462 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:28.211464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:28.211841 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:28.211875 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:28.211977 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:28.212481 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:28.212494 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:28.214270 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
W20260128 15:03:28.214543 23597 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:28.209820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:44125 (local address 127.13.88.190:39313)
0128 15:03:28.209955 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:28.209960 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:28.210088 (+ 128us) server_negotiation.cc:408] Connection header received
0128 15:03:28.210196 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:28.210199 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:28.210244 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:28.210308 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:28.210880 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:28.211372 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:28.211992 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:28.212120 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:28.214427 (+ 2307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:44125: BlockingRecv error: recv got EOF from 127.13.88.131:44125 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jan 28 15:03:28 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:28 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:28.872592 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:28.867638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:42671)
0128 15:03:28.867947 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:28.867960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:28.868055 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:28.868343 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:28.868346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:28.868581 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:28.868789 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:28.868796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:28.869621 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:28.869624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:28.870025 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:28.870031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:28.870128 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:28.870662 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:28.870676 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:28.872390 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":83,"threads_started":1}
W20260128 15:03:28.872767 23601 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:28.867768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:42671 (local address 127.13.88.190:39313)
0128 15:03:28.868034 (+ 266us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:28.868038 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:28.868052 (+ 14us) server_negotiation.cc:408] Connection header received
0128 15:03:28.868136 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:28.868139 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:28.868186 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:28.868271 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:28.868923 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:28.869496 (+ 573us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:28.870144 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:28.870303 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:28.872620 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:42671: BlockingRecv error: recv got EOF from 127.13.88.129:42671 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":68,"threads_started":1}
Jan 28 15:03:29 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:29 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:29.046247 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:29.041505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:38203)
0128 15:03:29.041935 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:29.041948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:29.042049 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:29.042294 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:29.042297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:29.042506 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:29.042737 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:29.042745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:29.043433 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:29.043435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:29.043822 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:29.043828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:29.043917 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:29.044379 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:29.044392 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:29.046131 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":82,"threads_started":1}
W20260128 15:03:29.046360 23601 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:29.041638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:38203 (local address 127.13.88.190:39313)
0128 15:03:29.041787 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:29.041792 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:29.042022 (+ 230us) server_negotiation.cc:408] Connection header received
0128 15:03:29.042134 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:29.042136 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:29.042184 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:29.042253 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:29.042848 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:29.043345 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:29.043904 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:29.044032 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:29.046260 (+ 2228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:38203: BlockingRecv error: recv got EOF from 127.13.88.130:38203 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jan 28 15:03:29 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:29 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.131@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:29.220297 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:29.215207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.131:34621)
0128 15:03:29.215516 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:29.215533 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:29.215681 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:29.215934 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:29.215937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:29.216137 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:29.216344 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:29.216351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:29.217108 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:29.217111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:29.217502 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:29.217509 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:29.217618 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:29.218194 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:29.218207 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:29.220150 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":94,"threads_started":1}
W20260128 15:03:29.220443 23601 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:29.215331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:34621 (local address 127.13.88.190:39313)
0128 15:03:29.215516 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:29.215521 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:29.215614 (+ 93us) server_negotiation.cc:408] Connection header received
0128 15:03:29.215776 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:29.215780 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:29.215828 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:29.215894 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:29.216501 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:29.216996 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:29.217610 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:29.217789 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:29.220293 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.131:34621: BlockingRecv error: recv got EOF from 127.13.88.131:34621 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jan 28 15:03:29 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:29 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.129@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:29.879024 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:29.873665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.129:56429)
0128 15:03:29.873952 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:29.873966 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:29.874053 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:29.874278 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:29.874280 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:29.874472 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:29.874694 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:29.874701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:29.875432 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:29.875435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:29.875914 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:29.875925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:29.876047 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:29.876702 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:29.876719 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:29.878865 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":85,"threads_started":1}
W20260128 15:03:29.879565 23605 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:29.873755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:56429 (local address 127.13.88.190:39313)
0128 15:03:29.874001 (+ 246us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:29.874005 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:29.874017 (+ 12us) server_negotiation.cc:408] Connection header received
0128 15:03:29.874130 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:29.874132 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:29.874177 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:29.874253 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:29.874816 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:29.875346 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:29.876048 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:29.876263 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:29.879432 (+ 3169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.129:56429: BlockingRecv error: recv got EOF from 127.13.88.129:56429 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":79,"threads_started":1}
Jan 28 15:03:30 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
Jan 28 15:03:30 dist-test-slave-m28t krb5kdc[22538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.88.130@KRBTEST.COM for oryx/127.13.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260128 15:03:30.051327 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:30.047023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:39313 (local address 127.13.88.130:39357)
0128 15:03:30.047275 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:30.047288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:30.047382 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:30.047589 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:30.047592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:30.047761 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:30.047948 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:30.047953 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.048640 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:30.048643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:30.049017 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:30.049023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.049109 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:30.049587 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:30.049600 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:30.051204 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.88.190:39313: Server oryx/127.13.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
W20260128 15:03:30.051425 23605 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:30.047123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:39357 (local address 127.13.88.190:39313)
0128 15:03:30.047278 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:30.047282 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:30.047349 (+ 67us) server_negotiation.cc:408] Connection header received
0128 15:03:30.047455 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:30.047458 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:30.047503 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:30.047566 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:30.048038 (+ 472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.048563 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:30.049096 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.049218 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:30.051330 (+ 2112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.88.130:39357: BlockingRecv error: recv got EOF from 127.13.88.130:39357 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260128 15:03:30.138108 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 23130
I20260128 15:03:30.142963 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 23282
I20260128 15:03:30.147635 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 23414
I20260128 15:03:30.152669 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 23060
2026-01-28T15:03:30Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18731 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-01-28T15:03:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:03:30Z Disabled control of system clock
I20260128 15:03:30.180208 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:45067
--webserver_interface=127.13.88.190
--webserver_port=0
--builtin_ntp_servers=127.13.88.148:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:45067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260128 15:03:30.286926 23610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:30.287150 23610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:30.287196 23610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:30.290462 23610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:03:30.290525 23610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:30.290540 23610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:30.290557 23610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:03:30.290578 23610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:03:30.294322 23610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:45067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:45067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:30.295382 23610 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:30.296561 23610 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:30.302518 23615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:30.302520 23616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:30.302577 23610 server_base.cc:1060] running on GCE node
W20260128 15:03:30.302603 23618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:30.303205 23610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:30.304082 23610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:30.305248 23610 hybrid_clock.cc:648] HybridClock initialized: now 1769612610305237 us; error 28 us; skew 500 ppm
I20260128 15:03:30.306953 23610 webserver.cc:492] Webserver started at http://127.13.88.190:37505/ using document root <none> and password file <none>
I20260128 15:03:30.307484 23610 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:30.307549 23610 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:30.307718 23610 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:30.309365 23610 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "b864f814a6b147c49234d005d2d84f1d"
format_stamp: "Formatted at 2026-01-28 15:03:30 on dist-test-slave-m28t"
server_key: "f283af5b8e4e17e35f23f0b379efe24e"
server_key_iv: "a78c089a77bfab5557cd7eb36122f2f5"
server_key_version: "encryptionkey@0"
I20260128 15:03:30.309811 23610 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "b864f814a6b147c49234d005d2d84f1d"
format_stamp: "Formatted at 2026-01-28 15:03:30 on dist-test-slave-m28t"
server_key: "f283af5b8e4e17e35f23f0b379efe24e"
server_key_iv: "a78c089a77bfab5557cd7eb36122f2f5"
server_key_version: "encryptionkey@0"
I20260128 15:03:30.312960 23610 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260128 15:03:30.314986 23624 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:30.315932 23610 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:03:30.316042 23610 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "b864f814a6b147c49234d005d2d84f1d"
format_stamp: "Formatted at 2026-01-28 15:03:30 on dist-test-slave-m28t"
server_key: "f283af5b8e4e17e35f23f0b379efe24e"
server_key_iv: "a78c089a77bfab5557cd7eb36122f2f5"
server_key_version: "encryptionkey@0"
I20260128 15:03:30.316125 23610 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:30.326932 23610 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:30.327546 23610 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:30.327706 23610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:30.334322 23676 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:45067 every 8 connection(s)
I20260128 15:03:30.334326 23610 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:45067
I20260128 15:03:30.335355 23610 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:03:30.338048 23677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:30.343226 23677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d: Bootstrap starting.
I20260128 15:03:30.345310 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 23610
I20260128 15:03:30.345395 23677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:30.345423 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:03:30.345638 13666 external_mini_cluster.cc:1468] Setting key d8a98571a4643dc97509da9953c5c864
I20260128 15:03:30.346082 23677 log.cc:826] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d: Log is configured to *not* fsync() on all Append() calls
I20260128 15:03:30.347870 23677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d: No bootstrap required, opened a new log
I20260128 15:03:30.350585 23677 raft_consensus.cc:359] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b864f814a6b147c49234d005d2d84f1d" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 45067 } }
I20260128 15:03:30.350770 23677 raft_consensus.cc:385] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:30.350834 23677 raft_consensus.cc:740] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b864f814a6b147c49234d005d2d84f1d, State: Initialized, Role: FOLLOWER
I20260128 15:03:30.351256 23677 consensus_queue.cc:260] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b864f814a6b147c49234d005d2d84f1d" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 45067 } }
I20260128 15:03:30.351374 23677 raft_consensus.cc:399] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:03:30.351428 23677 raft_consensus.cc:493] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:03:30.351516 23677 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:30.352384 23677 raft_consensus.cc:515] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b864f814a6b147c49234d005d2d84f1d" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 45067 } }
I20260128 15:03:30.352679 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:30.346588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.13.88.190:45067)
0128 15:03:30.347128 (+ 540us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:30.347137 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:30.347170 (+ 33us) server_negotiation.cc:408] Connection header received
0128 15:03:30.347684 (+ 514us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:30.347700 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:30.347973 (+ 273us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:30.348244 (+ 271us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:30.348774 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.349499 (+ 725us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:30.350309 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.350511 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:30.351091 (+ 580us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:30.351112 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:30.351124 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:03:30.351561 (+ 437us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:30.351617 (+ 56us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:03:30.351634 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:30.351711 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:30.351858 (+ 147us) server_negotiation.cc:300] Negotiation successful
0128 15:03:30.352075 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":103,"threads_started":1}
I20260128 15:03:30.352722 23677 leader_election.cc:304] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b864f814a6b147c49234d005d2d84f1d; no voters:
I20260128 15:03:30.352991 23677 leader_election.cc:290] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:03:30.353068 23682 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:03:30.353242 23682 raft_consensus.cc:697] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [term 1 LEADER]: Becoming Leader. State: Replica: b864f814a6b147c49234d005d2d84f1d, State: Running, Role: LEADER
I20260128 15:03:30.353586 23682 consensus_queue.cc:237] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b864f814a6b147c49234d005d2d84f1d" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 45067 } }
I20260128 15:03:30.354030 23677 sys_catalog.cc:565] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:03:30.354650 23684 sys_catalog.cc:455] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b864f814a6b147c49234d005d2d84f1d. Latest consensus state: current_term: 1 leader_uuid: "b864f814a6b147c49234d005d2d84f1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b864f814a6b147c49234d005d2d84f1d" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 45067 } } }
I20260128 15:03:30.354744 23684 sys_catalog.cc:458] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:30.355465 23683 sys_catalog.cc:455] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b864f814a6b147c49234d005d2d84f1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b864f814a6b147c49234d005d2d84f1d" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 45067 } } }
I20260128 15:03:30.355553 23683 sys_catalog.cc:458] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d [sys.catalog]: This master's current role is: LEADER
W20260128 15:03:30.358186 23697 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260128 15:03:30.358251 23697 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260128 15:03:30.358331 23689 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:03:30.359221 23689 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:03:30.363896 23689 catalog_manager.cc:1357] Generated new cluster ID: 7d45ddcbb6d54ca79608a90cda38dff9
I20260128 15:03:30.363957 23689 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:03:30.376386 23689 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:03:30.377290 23689 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:03:30.384687 23689 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b864f814a6b147c49234d005d2d84f1d: Generated new TSK 0
I20260128 15:03:30.385370 23689 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260128 15:03:30.390190 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:45067
--builtin_ntp_servers=127.13.88.148:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260128 15:03:30.493049 23701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:30.493269 23701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:30.493315 23701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:30.496622 23701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:30.496678 23701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:30.496752 23701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:03:30.500677 23701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:45067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:30.502138 23701 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:30.503211 23701 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:30.509341 23706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:30.509366 23709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:30.509572 23707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:30.509568 23701 server_base.cc:1060] running on GCE node
I20260128 15:03:30.509940 23701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:30.510448 23701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:30.511655 23701 hybrid_clock.cc:648] HybridClock initialized: now 1769612610511629 us; error 44 us; skew 500 ppm
I20260128 15:03:30.513445 23701 webserver.cc:492] Webserver started at http://127.13.88.129:45879/ using document root <none> and password file <none>
I20260128 15:03:30.513975 23701 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:30.514034 23701 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:30.514206 23701 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:30.515899 23701 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "8302d2e72baa4777bf2fe632e24e761f"
format_stamp: "Formatted at 2026-01-28 15:03:30 on dist-test-slave-m28t"
server_key: "6f817adfb0667583f89bc97ea3e96209"
server_key_iv: "9bd902c24c4685a7f517401e35a03da6"
server_key_version: "encryptionkey@0"
I20260128 15:03:30.516347 23701 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "8302d2e72baa4777bf2fe632e24e761f"
format_stamp: "Formatted at 2026-01-28 15:03:30 on dist-test-slave-m28t"
server_key: "6f817adfb0667583f89bc97ea3e96209"
server_key_iv: "9bd902c24c4685a7f517401e35a03da6"
server_key_version: "encryptionkey@0"
I20260128 15:03:30.519543 23701 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260128 15:03:30.521739 23715 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:30.522723 23701 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260128 15:03:30.522815 23701 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "8302d2e72baa4777bf2fe632e24e761f"
format_stamp: "Formatted at 2026-01-28 15:03:30 on dist-test-slave-m28t"
server_key: "6f817adfb0667583f89bc97ea3e96209"
server_key_iv: "9bd902c24c4685a7f517401e35a03da6"
server_key_version: "encryptionkey@0"
I20260128 15:03:30.522907 23701 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:30.542644 23701 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:30.543288 23701 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:30.543458 23701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:30.544009 23701 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:30.545032 23701 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:30.545091 23701 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:30.545138 23701 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:30.545167 23701 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:30.553678 23701 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:37779
I20260128 15:03:30.553718 23828 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:37779 every 8 connection(s)
I20260128 15:03:30.554622 23701 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:03:30.555373 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 23701
I20260128 15:03:30.555528 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:03:30.555794 13666 external_mini_cluster.cc:1468] Setting key 45ab50f59a4c5fa9d2b1e35489c34823
I20260128 15:03:30.557823 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:45067
--builtin_ntp_servers=127.13.88.148:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260128 15:03:30.562939 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:30.556459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:40095 (local address 127.13.88.190:45067)
0128 15:03:30.556610 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:30.556615 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:30.557751 (+ 1136us) server_negotiation.cc:408] Connection header received
0128 15:03:30.558749 (+ 998us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:30.558752 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:30.558801 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:30.558872 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:30.559925 (+ 1053us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.560480 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:30.561114 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.561288 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:30.562460 (+ 1172us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:30.562480 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:30.562483 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:03:30.562496 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:30.562517 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:03:30.562524 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:30.562583 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:30.562735 (+ 152us) server_negotiation.cc:300] Negotiation successful
0128 15:03:30.562801 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260128 15:03:30.563575 23831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:30.556826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:45067 (local address 127.13.88.129:40095)
0128 15:03:30.557591 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:30.557648 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:30.558547 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:30.559034 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:30.559045 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:30.559149 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:30.559797 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:30.559809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.560605 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:30.560608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:30.561010 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:30.561017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.561222 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:30.562240 (+ 1018us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 15:03:30.562241 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:30.562276 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 15:03:30.562327 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:30.562598 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:30.562606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:30.562610 (+ 4us) client_negotiation.cc:769] Sending connection context
0128 15:03:30.562741 (+ 131us) client_negotiation.cc:240] Negotiation successful
0128 15:03:30.562965 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":135,"threads_started":1}
I20260128 15:03:30.564688 23829 heartbeater.cc:344] Connected to a master server at 127.13.88.190:45067
I20260128 15:03:30.564960 23829 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:30.565649 23829 heartbeater.cc:507] Master 127.13.88.190:45067 requested a full tablet report, sending...
I20260128 15:03:30.567299 23641 ts_manager.cc:194] Registered new tserver with Master: 8302d2e72baa4777bf2fe632e24e761f (127.13.88.129:37779)
I20260128 15:03:30.568979 23641 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.88.129:40095
W20260128 15:03:30.663233 23832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:30.663458 23832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:30.663504 23832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:30.666719 23832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:30.666770 23832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:30.666836 23832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:03:30.670468 23832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:45067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:30.671444 23832 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:30.672505 23832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:30.678773 23837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:30.678825 23840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:30.678773 23838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:30.678965 23832 server_base.cc:1060] running on GCE node
I20260128 15:03:30.679406 23832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:30.679946 23832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:30.681102 23832 hybrid_clock.cc:648] HybridClock initialized: now 1769612610681090 us; error 25 us; skew 500 ppm
I20260128 15:03:30.682830 23832 webserver.cc:492] Webserver started at http://127.13.88.130:43601/ using document root <none> and password file <none>
I20260128 15:03:30.683357 23832 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:30.683413 23832 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:30.683590 23832 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:30.685240 23832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "778622a019c44fc5b9cf4d63e4570b92"
format_stamp: "Formatted at 2026-01-28 15:03:30 on dist-test-slave-m28t"
server_key: "fce87fefbfcd019199a09316f58e56ca"
server_key_iv: "d345debce49cdd85be0d53d477373ea7"
server_key_version: "encryptionkey@0"
I20260128 15:03:30.685676 23832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "778622a019c44fc5b9cf4d63e4570b92"
format_stamp: "Formatted at 2026-01-28 15:03:30 on dist-test-slave-m28t"
server_key: "fce87fefbfcd019199a09316f58e56ca"
server_key_iv: "d345debce49cdd85be0d53d477373ea7"
server_key_version: "encryptionkey@0"
I20260128 15:03:30.688776 23832 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260128 15:03:30.690785 23846 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:30.691738 23832 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:03:30.691838 23832 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "778622a019c44fc5b9cf4d63e4570b92"
format_stamp: "Formatted at 2026-01-28 15:03:30 on dist-test-slave-m28t"
server_key: "fce87fefbfcd019199a09316f58e56ca"
server_key_iv: "d345debce49cdd85be0d53d477373ea7"
server_key_version: "encryptionkey@0"
I20260128 15:03:30.691920 23832 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:30.714053 23832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:30.714740 23832 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:30.714907 23832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:30.715445 23832 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:30.716476 23832 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:30.716526 23832 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:30.716578 23832 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:30.716603 23832 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:30.725274 23832 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:32807
I20260128 15:03:30.725301 23959 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:32807 every 8 connection(s)
I20260128 15:03:30.726251 23832 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
I20260128 15:03:30.733592 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 23832
I20260128 15:03:30.733708 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:03:30.733934 13666 external_mini_cluster.cc:1468] Setting key d6c255c595e72bbbb38ab93cdfa47ce0
I20260128 15:03:30.734555 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:30.727973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:35043 (local address 127.13.88.190:45067)
0128 15:03:30.728121 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:30.728126 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:30.729140 (+ 1014us) server_negotiation.cc:408] Connection header received
0128 15:03:30.730003 (+ 863us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:30.730007 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:30.730063 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:30.730146 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:30.731185 (+ 1039us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.731726 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:30.732490 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.732640 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:30.733930 (+ 1290us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:30.733951 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:30.733953 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:03:30.733968 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:30.733990 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:03:30.733998 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:30.734075 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:30.734301 (+ 226us) server_negotiation.cc:300] Negotiation successful
0128 15:03:30.734380 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260128 15:03:30.735275 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:30.728322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:45067 (local address 127.13.88.130:35043)
0128 15:03:30.728998 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:30.729035 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:30.729774 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:30.730302 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:30.730310 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:30.730401 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:30.731031 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:30.731043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.731886 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:30.731889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:30.732328 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:30.732336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.732590 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:30.733640 (+ 1050us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 15:03:30.733642 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:30.733689 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 15:03:30.733783 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:30.734094 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:30.734105 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:30.734111 (+ 6us) client_negotiation.cc:769] Sending connection context
0128 15:03:30.734296 (+ 185us) client_negotiation.cc:240] Negotiation successful
0128 15:03:30.734546 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":147,"threads_started":1}
I20260128 15:03:30.736021 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:45067
--builtin_ntp_servers=127.13.88.148:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260128 15:03:30.736331 23960 heartbeater.cc:344] Connected to a master server at 127.13.88.190:45067
I20260128 15:03:30.736625 23960 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:30.737203 23960 heartbeater.cc:507] Master 127.13.88.190:45067 requested a full tablet report, sending...
I20260128 15:03:30.738274 23641 ts_manager.cc:194] Registered new tserver with Master: 778622a019c44fc5b9cf4d63e4570b92 (127.13.88.130:32807)
I20260128 15:03:30.738852 23641 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.88.130:35043
W20260128 15:03:30.841665 23963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:30.841912 23963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:30.841979 23963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:30.845364 23963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:30.845434 23963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:30.845530 23963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:03:30.849467 23963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:45067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:30.850595 23963 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:30.851766 23963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:30.858472 23968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:30.858479 23971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:30.858484 23969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:30.858799 23963 server_base.cc:1060] running on GCE node
I20260128 15:03:30.859171 23963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:30.859714 23963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:30.860852 23963 hybrid_clock.cc:648] HybridClock initialized: now 1769612610860832 us; error 35 us; skew 500 ppm
I20260128 15:03:30.862615 23963 webserver.cc:492] Webserver started at http://127.13.88.131:39817/ using document root <none> and password file <none>
I20260128 15:03:30.863169 23963 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:30.863221 23963 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:30.863384 23963 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:30.865234 23963 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "afdeb20a03f74816911a6e5f86be5dd3"
format_stamp: "Formatted at 2026-01-28 15:03:30 on dist-test-slave-m28t"
server_key: "224da5a635140f7401a04d9c2a8833ca"
server_key_iv: "87d0756fb3d509f38b976d413f39291a"
server_key_version: "encryptionkey@0"
I20260128 15:03:30.865717 23963 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "afdeb20a03f74816911a6e5f86be5dd3"
format_stamp: "Formatted at 2026-01-28 15:03:30 on dist-test-slave-m28t"
server_key: "224da5a635140f7401a04d9c2a8833ca"
server_key_iv: "87d0756fb3d509f38b976d413f39291a"
server_key_version: "encryptionkey@0"
I20260128 15:03:30.869141 23963 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260128 15:03:30.871340 23977 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:30.872359 23963 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:03:30.872530 23963 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "afdeb20a03f74816911a6e5f86be5dd3"
format_stamp: "Formatted at 2026-01-28 15:03:30 on dist-test-slave-m28t"
server_key: "224da5a635140f7401a04d9c2a8833ca"
server_key_iv: "87d0756fb3d509f38b976d413f39291a"
server_key_version: "encryptionkey@0"
I20260128 15:03:30.872630 23963 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:30.896010 23963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:30.896795 23963 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:30.896965 23963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:30.897531 23963 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:30.898543 23963 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:30.898589 23963 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:30.898633 23963 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:30.898661 23963 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:30.906967 23963 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:45541
I20260128 15:03:30.906997 24090 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:45541 every 8 connection(s)
I20260128 15:03:30.907938 23963 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
I20260128 15:03:30.911921 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 23963
I20260128 15:03:30.912012 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:03:30.912230 13666 external_mini_cluster.cc:1468] Setting key 08678f8c1f3e255e2b8a67b600a219e0
I20260128 15:03:30.916194 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:30.909637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:55361 (local address 127.13.88.190:45067)
0128 15:03:30.909783 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:30.909786 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:30.910682 (+ 896us) server_negotiation.cc:408] Connection header received
0128 15:03:30.911571 (+ 889us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:30.911575 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:30.911636 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:30.911853 (+ 217us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:30.912985 (+ 1132us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.913668 (+ 683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:30.914341 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.914528 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:30.915727 (+ 1199us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:30.915742 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:30.915745 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:03:30.915756 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:30.915775 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:03:30.915782 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:30.915843 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:30.915994 (+ 151us) server_negotiation.cc:300] Negotiation successful
0128 15:03:30.916055 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260128 15:03:30.916970 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:30.909907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:45067 (local address 127.13.88.131:55361)
0128 15:03:30.910504 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:30.910538 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:30.911360 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:30.912041 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:30.912053 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:30.912136 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:30.912851 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:30.912866 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.913803 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:30.913806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:30.914229 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:30.914237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:30.914432 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:30.915481 (+ 1049us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 15:03:30.915483 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:30.915516 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 15:03:30.915570 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:30.915856 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:30.915864 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:30.915869 (+ 5us) client_negotiation.cc:769] Sending connection context
0128 15:03:30.915994 (+ 125us) client_negotiation.cc:240] Negotiation successful
0128 15:03:30.916221 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":98,"threads_started":1}
I20260128 15:03:30.918107 24091 heartbeater.cc:344] Connected to a master server at 127.13.88.190:45067
I20260128 15:03:30.918326 24091 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:30.918848 24091 heartbeater.cc:507] Master 127.13.88.190:45067 requested a full tablet report, sending...
I20260128 15:03:30.919800 23641 ts_manager.cc:194] Registered new tserver with Master: afdeb20a03f74816911a6e5f86be5dd3 (127.13.88.131:45541)
I20260128 15:03:30.920315 23641 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.13.88.131:55361
I20260128 15:03:30.925706 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260128 15:03:30.928284 23681 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:30.927474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.13.88.190:45067)
0128 15:03:30.927597 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:30.927602 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:30.927691 (+ 89us) server_negotiation.cc:408] Connection header received
0128 15:03:30.927833 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:30.927837 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:30.927884 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:30.927942 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:30.928166 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36690: BlockingRecv error: recv got EOF from 127.0.0.1:36690 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260128 15:03:30.928401 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 23701
I20260128 15:03:30.933485 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 23832
I20260128 15:03:30.938097 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 23963
I20260128 15:03:30.942694 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 23610
2026-01-28T15:03:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (788 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-01-28T15:03:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:03:30Z Disabled control of system clock
I20260128 15:03:30.969318 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:37653
--webserver_interface=127.13.88.190
--webserver_port=0
--builtin_ntp_servers=127.13.88.148:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:37653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260128 15:03:31.076490 24102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:31.076725 24102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:31.076773 24102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:31.080009 24102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:03:31.080081 24102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:31.080096 24102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:31.080114 24102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:03:31.080132 24102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:03:31.084079 24102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:37653
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:37653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:31.085156 24102 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:31.086210 24102 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:31.091728 24110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:31.091732 24107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:31.091909 24102 server_base.cc:1060] running on GCE node
W20260128 15:03:31.091732 24108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:31.092567 24102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:31.093454 24102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:31.094609 24102 hybrid_clock.cc:648] HybridClock initialized: now 1769612611094590 us; error 37 us; skew 500 ppm
I20260128 15:03:31.096392 24102 webserver.cc:492] Webserver started at http://127.13.88.190:36517/ using document root <none> and password file <none>
I20260128 15:03:31.097007 24102 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:31.097067 24102 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:31.097275 24102 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:31.098987 24102 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "545f350c70e840ba8586be2bff6fb409"
format_stamp: "Formatted at 2026-01-28 15:03:31 on dist-test-slave-m28t"
server_key: "5873193f700e959fe0fe1473ad3787ca"
server_key_iv: "48c6ff3a679b56f27c2bcaa9d201c823"
server_key_version: "encryptionkey@0"
I20260128 15:03:31.099426 24102 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "545f350c70e840ba8586be2bff6fb409"
format_stamp: "Formatted at 2026-01-28 15:03:31 on dist-test-slave-m28t"
server_key: "5873193f700e959fe0fe1473ad3787ca"
server_key_iv: "48c6ff3a679b56f27c2bcaa9d201c823"
server_key_version: "encryptionkey@0"
I20260128 15:03:31.102665 24102 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260128 15:03:31.104943 24116 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:31.105952 24102 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260128 15:03:31.106058 24102 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "545f350c70e840ba8586be2bff6fb409"
format_stamp: "Formatted at 2026-01-28 15:03:31 on dist-test-slave-m28t"
server_key: "5873193f700e959fe0fe1473ad3787ca"
server_key_iv: "48c6ff3a679b56f27c2bcaa9d201c823"
server_key_version: "encryptionkey@0"
I20260128 15:03:31.106145 24102 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:31.110808 24102 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:31.111344 24102 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:31.111493 24102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:31.118314 24168 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:37653 every 8 connection(s)
I20260128 15:03:31.118326 24102 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:37653
I20260128 15:03:31.119277 24102 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:03:31.122074 24169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:31.124302 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 24102
I20260128 15:03:31.124468 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:03:31.124775 13666 external_mini_cluster.cc:1468] Setting key 725933155a24bfb5cad43e59871dade0
I20260128 15:03:31.128211 24169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409: Bootstrap starting.
I20260128 15:03:31.130419 24169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:31.130692 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:31.127069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42662 (local address 127.13.88.190:37653)
0128 15:03:31.127684 (+ 615us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:31.127692 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:31.127728 (+ 36us) server_negotiation.cc:408] Connection header received
0128 15:03:31.128219 (+ 491us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:31.128235 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:31.128609 (+ 374us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:31.128906 (+ 297us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:31.129639 (+ 733us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:31.129644 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:31.129654 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:03:31.129663 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:31.129705 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:03:31.129715 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:31.129842 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:31.129927 (+ 85us) server_negotiation.cc:300] Negotiation successful
0128 15:03:31.130139 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":123,"threads_started":1}
I20260128 15:03:31.131120 24169 log.cc:826] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409: Log is configured to *not* fsync() on all Append() calls
I20260128 15:03:31.132861 24169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409: No bootstrap required, opened a new log
I20260128 15:03:31.135224 24169 raft_consensus.cc:359] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545f350c70e840ba8586be2bff6fb409" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 37653 } }
I20260128 15:03:31.135558 24169 raft_consensus.cc:385] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:31.135648 24169 raft_consensus.cc:740] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545f350c70e840ba8586be2bff6fb409, State: Initialized, Role: FOLLOWER
I20260128 15:03:31.136056 24169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545f350c70e840ba8586be2bff6fb409" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 37653 } }
I20260128 15:03:31.136178 24169 raft_consensus.cc:399] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:03:31.136234 24169 raft_consensus.cc:493] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:03:31.136318 24169 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:31.137151 24169 raft_consensus.cc:515] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545f350c70e840ba8586be2bff6fb409" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 37653 } }
I20260128 15:03:31.137467 24169 leader_election.cc:304] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 545f350c70e840ba8586be2bff6fb409; no voters:
I20260128 15:03:31.137732 24169 leader_election.cc:290] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:03:31.137830 24174 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:03:31.138007 24174 raft_consensus.cc:697] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [term 1 LEADER]: Becoming Leader. State: Replica: 545f350c70e840ba8586be2bff6fb409, State: Running, Role: LEADER
I20260128 15:03:31.138331 24174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545f350c70e840ba8586be2bff6fb409" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 37653 } }
I20260128 15:03:31.138744 24169 sys_catalog.cc:565] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:03:31.139755 24175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "545f350c70e840ba8586be2bff6fb409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545f350c70e840ba8586be2bff6fb409" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 37653 } } }
I20260128 15:03:31.139865 24175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:31.140174 24176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 545f350c70e840ba8586be2bff6fb409. Latest consensus state: current_term: 1 leader_uuid: "545f350c70e840ba8586be2bff6fb409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545f350c70e840ba8586be2bff6fb409" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 37653 } } }
I20260128 15:03:31.140307 24176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409 [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:31.140385 24183 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:03:31.143461 24183 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:03:31.148712 24183 catalog_manager.cc:1357] Generated new cluster ID: 7a8a3f4f65e54aafbfdbccfc2095bed1
I20260128 15:03:31.148788 24183 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:03:31.171703 24183 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:03:31.172019 24183 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:03:31.179111 24183 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 545f350c70e840ba8586be2bff6fb409: Generated new TSK 0
I20260128 15:03:31.179823 24183 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260128 15:03:31.183656 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:37653
--builtin_ntp_servers=127.13.88.148:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260128 15:03:31.287628 24193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:31.287842 24193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:31.287909 24193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:31.291172 24193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:31.291229 24193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:31.291332 24193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:03:31.295238 24193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:37653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:31.296379 24193 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:31.297577 24193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:31.304054 24198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:31.304193 24193 server_base.cc:1060] running on GCE node
W20260128 15:03:31.304246 24201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:31.304054 24199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:31.304730 24193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:31.305323 24193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:31.306479 24193 hybrid_clock.cc:648] HybridClock initialized: now 1769612611306457 us; error 38 us; skew 500 ppm
I20260128 15:03:31.308485 24193 webserver.cc:492] Webserver started at http://127.13.88.129:45083/ using document root <none> and password file <none>
I20260128 15:03:31.309130 24193 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:31.309194 24193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:31.309383 24193 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:31.311514 24193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "3d68d2599239451bba932dda35e8ce9d"
format_stamp: "Formatted at 2026-01-28 15:03:31 on dist-test-slave-m28t"
server_key: "5a16a776f0f6eef0db7df3411bf00350"
server_key_iv: "5082df7bfe8536481ddd0c36fd3efaf7"
server_key_version: "encryptionkey@0"
I20260128 15:03:31.312016 24193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "3d68d2599239451bba932dda35e8ce9d"
format_stamp: "Formatted at 2026-01-28 15:03:31 on dist-test-slave-m28t"
server_key: "5a16a776f0f6eef0db7df3411bf00350"
server_key_iv: "5082df7bfe8536481ddd0c36fd3efaf7"
server_key_version: "encryptionkey@0"
I20260128 15:03:31.315759 24193 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260128 15:03:31.317857 24207 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:31.318835 24193 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:03:31.318921 24193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "3d68d2599239451bba932dda35e8ce9d"
format_stamp: "Formatted at 2026-01-28 15:03:31 on dist-test-slave-m28t"
server_key: "5a16a776f0f6eef0db7df3411bf00350"
server_key_iv: "5082df7bfe8536481ddd0c36fd3efaf7"
server_key_version: "encryptionkey@0"
I20260128 15:03:31.318992 24193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:31.323355 24193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:31.323885 24193 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:31.324013 24193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:31.324498 24193 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:31.325381 24193 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:31.325431 24193 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:31.325469 24193 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:31.325487 24193 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:31.333729 24193 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:45867
I20260128 15:03:31.333777 24320 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:45867 every 8 connection(s)
I20260128 15:03:31.334838 24193 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:03:31.338832 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 24193
I20260128 15:03:31.338960 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:03:31.339208 13666 external_mini_cluster.cc:1468] Setting key 703c8d5cdadcc4daf157d96b31da297a
I20260128 15:03:31.340246 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:31.336671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:54163 (local address 127.13.88.190:37653)
0128 15:03:31.336806 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:31.336810 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:31.337738 (+ 928us) server_negotiation.cc:408] Connection header received
0128 15:03:31.338503 (+ 765us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:31.338507 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:31.338555 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:31.338630 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:31.339739 (+ 1109us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:31.339744 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:31.339747 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:03:31.339761 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:31.339812 (+ 51us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:03:31.339823 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:31.339907 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:31.340061 (+ 154us) server_negotiation.cc:300] Negotiation successful
0128 15:03:31.340099 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260128 15:03:31.341030 24323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:31.336966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:37653 (local address 127.13.88.129:54163)
0128 15:03:31.337600 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:31.337634 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:31.338314 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:31.338785 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:31.338793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:31.338876 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:31.339489 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 15:03:31.339491 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:31.339525 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 15:03:31.339577 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:31.339909 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:31.339919 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:31.339924 (+ 5us) client_negotiation.cc:769] Sending connection context
0128 15:03:31.340103 (+ 179us) client_negotiation.cc:240] Negotiation successful
0128 15:03:31.340332 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":96,"threads_started":1}
I20260128 15:03:31.341142 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:37653
--builtin_ntp_servers=127.13.88.148:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260128 15:03:31.342053 24321 heartbeater.cc:344] Connected to a master server at 127.13.88.190:37653
I20260128 15:03:31.342304 24321 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:31.342725 24321 heartbeater.cc:507] Master 127.13.88.190:37653 requested a full tablet report, sending...
I20260128 15:03:31.344136 24133 ts_manager.cc:194] Registered new tserver with Master: 3d68d2599239451bba932dda35e8ce9d (127.13.88.129:45867)
W20260128 15:03:31.449539 24324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:31.449791 24324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:31.449864 24324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:31.453253 24324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:31.453320 24324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:31.453423 24324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:03:31.457336 24324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:37653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:31.458415 24324 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:31.459539 24324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:31.466152 24329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:31.466153 24332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:31.466153 24330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:31.466238 24324 server_base.cc:1060] running on GCE node
I20260128 15:03:31.466835 24324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:31.467361 24324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:31.468494 24324 hybrid_clock.cc:648] HybridClock initialized: now 1769612611468466 us; error 37 us; skew 500 ppm
I20260128 15:03:31.470371 24324 webserver.cc:492] Webserver started at http://127.13.88.130:38829/ using document root <none> and password file <none>
I20260128 15:03:31.470944 24324 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:31.470999 24324 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:31.471184 24324 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:31.472976 24324 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "b05162b6df01483aa4fca866a1979434"
format_stamp: "Formatted at 2026-01-28 15:03:31 on dist-test-slave-m28t"
server_key: "f82b6e30c52bf8e7b38c01fe582588b8"
server_key_iv: "bc068f523315e513e55243858f5d9aa8"
server_key_version: "encryptionkey@0"
I20260128 15:03:31.473438 24324 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "b05162b6df01483aa4fca866a1979434"
format_stamp: "Formatted at 2026-01-28 15:03:31 on dist-test-slave-m28t"
server_key: "f82b6e30c52bf8e7b38c01fe582588b8"
server_key_iv: "bc068f523315e513e55243858f5d9aa8"
server_key_version: "encryptionkey@0"
I20260128 15:03:31.476831 24324 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260128 15:03:31.479029 24338 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:31.480074 24324 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:03:31.480177 24324 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "b05162b6df01483aa4fca866a1979434"
format_stamp: "Formatted at 2026-01-28 15:03:31 on dist-test-slave-m28t"
server_key: "f82b6e30c52bf8e7b38c01fe582588b8"
server_key_iv: "bc068f523315e513e55243858f5d9aa8"
server_key_version: "encryptionkey@0"
I20260128 15:03:31.480265 24324 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:31.484992 24324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:31.485545 24324 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:31.485694 24324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:31.486191 24324 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:31.487115 24324 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:31.487160 24324 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:31.487196 24324 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:31.487217 24324 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:31.495496 24324 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:37831
I20260128 15:03:31.495527 24451 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:37831 every 8 connection(s)
I20260128 15:03:31.496492 24324 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
I20260128 15:03:31.501641 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:31.498196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:34875 (local address 127.13.88.190:37653)
0128 15:03:31.498327 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:31.498331 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:31.499189 (+ 858us) server_negotiation.cc:408] Connection header received
0128 15:03:31.500009 (+ 820us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:31.500013 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:31.500064 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:31.500142 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:31.501194 (+ 1052us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:31.501197 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:31.501200 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:03:31.501213 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:31.501232 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:03:31.501239 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:31.501321 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:31.501454 (+ 133us) server_negotiation.cc:300] Negotiation successful
0128 15:03:31.501503 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260128 15:03:31.502319 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:31.498432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:37653 (local address 127.13.88.130:34875)
0128 15:03:31.499041 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:31.499076 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:31.499807 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:31.500315 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:31.500323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:31.500409 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:31.500960 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 15:03:31.500962 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:31.500996 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 15:03:31.501051 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:31.501309 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:31.501316 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:31.501321 (+ 5us) client_negotiation.cc:769] Sending connection context
0128 15:03:31.501523 (+ 202us) client_negotiation.cc:240] Negotiation successful
0128 15:03:31.501740 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
I20260128 15:03:31.503402 24452 heartbeater.cc:344] Connected to a master server at 127.13.88.190:37653
I20260128 15:03:31.503626 24452 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:31.504022 24452 heartbeater.cc:507] Master 127.13.88.190:37653 requested a full tablet report, sending...
I20260128 15:03:31.505005 24133 ts_manager.cc:194] Registered new tserver with Master: b05162b6df01483aa4fca866a1979434 (127.13.88.130:37831)
I20260128 15:03:31.506791 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 24324
I20260128 15:03:31.506892 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:03:31.507103 13666 external_mini_cluster.cc:1468] Setting key d201441aef01d2cd99a62bd4720fa292
I20260128 15:03:31.508800 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:37653
--builtin_ntp_servers=127.13.88.148:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260128 15:03:31.613619 24455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:31.613829 24455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:31.613888 24455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:31.617131 24455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:31.617182 24455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:31.617269 24455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:03:31.621078 24455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:37653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:31.622107 24455 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:31.623250 24455 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:31.629698 24460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:31.629793 24455 server_base.cc:1060] running on GCE node
W20260128 15:03:31.629693 24463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:31.629698 24461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:31.630221 24455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:31.630764 24455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:31.631937 24455 hybrid_clock.cc:648] HybridClock initialized: now 1769612611631916 us; error 30 us; skew 500 ppm
I20260128 15:03:31.633652 24455 webserver.cc:492] Webserver started at http://127.13.88.131:37887/ using document root <none> and password file <none>
I20260128 15:03:31.634207 24455 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:31.634260 24455 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:31.634441 24455 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:31.636092 24455 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "976a0e0619404bea9836c0d7de505b58"
format_stamp: "Formatted at 2026-01-28 15:03:31 on dist-test-slave-m28t"
server_key: "5fda220b8dfe68f586aa5054faee65fa"
server_key_iv: "82bf565de10bad0ebcbd13a683f3a1de"
server_key_version: "encryptionkey@0"
I20260128 15:03:31.636590 24455 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "976a0e0619404bea9836c0d7de505b58"
format_stamp: "Formatted at 2026-01-28 15:03:31 on dist-test-slave-m28t"
server_key: "5fda220b8dfe68f586aa5054faee65fa"
server_key_iv: "82bf565de10bad0ebcbd13a683f3a1de"
server_key_version: "encryptionkey@0"
I20260128 15:03:31.639776 24455 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260128 15:03:31.641942 24469 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:31.642891 24455 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:03:31.642984 24455 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "976a0e0619404bea9836c0d7de505b58"
format_stamp: "Formatted at 2026-01-28 15:03:31 on dist-test-slave-m28t"
server_key: "5fda220b8dfe68f586aa5054faee65fa"
server_key_iv: "82bf565de10bad0ebcbd13a683f3a1de"
server_key_version: "encryptionkey@0"
I20260128 15:03:31.643071 24455 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:31.647637 24455 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:31.648165 24455 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:31.648293 24455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:31.648799 24455 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:31.649705 24455 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:31.649749 24455 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:31.649783 24455 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:31.649803 24455 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:31.658144 24455 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:44535
I20260128 15:03:31.658169 24582 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:44535 every 8 connection(s)
I20260128 15:03:31.659072 24455 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
I20260128 15:03:31.663648 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 24455
I20260128 15:03:31.663760 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:03:31.663980 13666 external_mini_cluster.cc:1468] Setting key 75f00821a7d442dfac807a7ed0c44fd0
I20260128 15:03:31.664115 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:31.660753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:60147 (local address 127.13.88.190:37653)
0128 15:03:31.660894 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:31.660898 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:31.661772 (+ 874us) server_negotiation.cc:408] Connection header received
0128 15:03:31.662557 (+ 785us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:31.662561 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:31.662607 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:31.662677 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:31.663649 (+ 972us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:31.663653 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:31.663656 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0128 15:03:31.663668 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:31.663691 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0128 15:03:31.663698 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:31.663774 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:31.663933 (+ 159us) server_negotiation.cc:300] Negotiation successful
0128 15:03:31.663981 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260128 15:03:31.664958 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:31.661058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:37653 (local address 127.13.88.131:60147)
0128 15:03:31.661643 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:31.661676 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:31.662349 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:31.662839 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:31.662847 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:31.662927 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:31.663430 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0128 15:03:31.663432 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:31.663463 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0128 15:03:31.663517 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:31.663774 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:31.663782 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:31.663787 (+ 5us) client_negotiation.cc:769] Sending connection context
0128 15:03:31.663976 (+ 189us) client_negotiation.cc:240] Negotiation successful
0128 15:03:31.664176 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":92,"threads_started":1}
I20260128 15:03:31.665891 24583 heartbeater.cc:344] Connected to a master server at 127.13.88.190:37653
I20260128 15:03:31.666100 24583 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:31.666467 24583 heartbeater.cc:507] Master 127.13.88.190:37653 requested a full tablet report, sending...
I20260128 15:03:31.667322 24133 ts_manager.cc:194] Registered new tserver with Master: 976a0e0619404bea9836c0d7de505b58 (127.13.88.131:44535)
I20260128 15:03:31.678598 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260128 15:03:31.681205 24172 negotiation.cc:337] Failed RPC negotiation. Trace:
0128 15:03:31.680450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42678 (local address 127.13.88.190:37653)
0128 15:03:31.680590 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:31.680595 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:31.680697 (+ 102us) server_negotiation.cc:408] Connection header received
0128 15:03:31.680813 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:31.680816 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:31.680858 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:31.680915 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:31.681083 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42678: BlockingRecv error: recv got EOF from 127.0.0.1:42678 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260128 15:03:31.681409 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 24193
I20260128 15:03:31.686472 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 24324
I20260128 15:03:31.691171 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 24455
I20260128 15:03:31.695819 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 24102
2026-01-28T15:03:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (753 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:03:31 dist-test-slave-m28t krb5kdc[24593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:03:31 dist-test-slave-m28t krb5kdc[24593](info): set up 2 sockets
Jan 28 15:03:31 dist-test-slave-m28t krb5kdc[24593](info): commencing operation
krb5kdc: starting...
W20260128 15:03:33.536906 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.821s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Jan 28 15:03:33 dist-test-slave-m28t krb5kdc[24593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:03:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:03:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:03:33.664326 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:40993
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:40993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:03:33.771457 24609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:33.771706 24609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:33.771760 24609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:33.775094 24609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:03:33.775163 24609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:33.775185 24609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:33.775209 24609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:03:33.775225 24609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:03:33.779493 24609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:40993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:40993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:33.780515 24609 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:33.781307 24609 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:33.786665 24614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:33.786671 24615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:33.786705 24617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:33.786748 24609 server_base.cc:1060] running on GCE node
I20260128 15:03:33.787416 24609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:33.788306 24609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:33.789456 24609 hybrid_clock.cc:648] HybridClock initialized: now 1769612613789437 us; error 35 us; skew 500 ppm
Jan 28 15:03:33 dist-test-slave-m28t krb5kdc[24593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612613, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:33.792143 24609 init.cc:377] Logged in from keytab as kudu/127.13.88.190@KRBTEST.COM (short username kudu)
I20260128 15:03:33.793220 24609 webserver.cc:492] Webserver started at http://127.13.88.190:45631/ using document root <none> and password file <none>
I20260128 15:03:33.793737 24609 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:33.793793 24609 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:33.793963 24609 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:33.795554 24609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "35872c210d044de596904d06b5937d61"
format_stamp: "Formatted at 2026-01-28 15:03:33 on dist-test-slave-m28t"
server_key: "f4357c0b1bd21fff4d8a3aad4e11142a"
server_key_iv: "8039491b650e005458250bc0cb50870d"
server_key_version: "encryptionkey@0"
I20260128 15:03:33.796000 24609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "35872c210d044de596904d06b5937d61"
format_stamp: "Formatted at 2026-01-28 15:03:33 on dist-test-slave-m28t"
server_key: "f4357c0b1bd21fff4d8a3aad4e11142a"
server_key_iv: "8039491b650e005458250bc0cb50870d"
server_key_version: "encryptionkey@0"
I20260128 15:03:33.799271 24609 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260128 15:03:33.801378 24624 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:33.802373 24609 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:03:33.802485 24609 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "35872c210d044de596904d06b5937d61"
format_stamp: "Formatted at 2026-01-28 15:03:33 on dist-test-slave-m28t"
server_key: "f4357c0b1bd21fff4d8a3aad4e11142a"
server_key_iv: "8039491b650e005458250bc0cb50870d"
server_key_version: "encryptionkey@0"
I20260128 15:03:33.802569 24609 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:33.812160 24609 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:33.818125 24609 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:33.818297 24609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:33.825340 24609 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:40993
I20260128 15:03:33.825343 24676 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:40993 every 8 connection(s)
I20260128 15:03:33.826331 24609 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:03:33.829121 24677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:33.829285 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 24609
I20260128 15:03:33.829396 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:03:33.829620 13666 external_mini_cluster.cc:1468] Setting key de1f562131f835d567a01087643b3e00
I20260128 15:03:33.835644 24677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61: Bootstrap starting.
Jan 28 15:03:33 dist-test-slave-m28t krb5kdc[24593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:03:33.838248 24677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:33.838953 24677 log.cc:826] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61: Log is configured to *not* fsync() on all Append() calls
I20260128 15:03:33.840749 24677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61: No bootstrap required, opened a new log
I20260128 15:03:33.842136 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:33.830761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.13.88.190:40993)
0128 15:03:33.831119 (+ 358us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:33.831128 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:33.831161 (+ 33us) server_negotiation.cc:408] Connection header received
0128 15:03:33.832211 (+ 1050us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:33.832231 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:33.832568 (+ 337us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:33.832897 (+ 329us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:33.833905 (+ 1008us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:33.834663 (+ 758us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:33.835528 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:33.835909 (+ 381us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:33.838077 (+ 2168us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:33.838095 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:33.838107 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:33.838149 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:33.839766 (+ 1617us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:33.840157 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:33.840162 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:33.840166 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:33.840239 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:33.840463 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:33.840467 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:33.840469 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:33.840818 (+ 349us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:33.840985 (+ 167us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:33.841275 (+ 290us) server_negotiation.cc:300] Negotiation successful
0128 15:03:33.841517 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
I20260128 15:03:33.843626 24677 raft_consensus.cc:359] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35872c210d044de596904d06b5937d61" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 40993 } }
I20260128 15:03:33.843815 24677 raft_consensus.cc:385] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:33.843885 24677 raft_consensus.cc:740] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35872c210d044de596904d06b5937d61, State: Initialized, Role: FOLLOWER
I20260128 15:03:33.844362 24677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35872c210d044de596904d06b5937d61" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 40993 } }
I20260128 15:03:33.844545 24677 raft_consensus.cc:399] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:03:33.844609 24677 raft_consensus.cc:493] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:03:33.844699 24677 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:33.845511 24677 raft_consensus.cc:515] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35872c210d044de596904d06b5937d61" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 40993 } }
I20260128 15:03:33.845835 24677 leader_election.cc:304] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35872c210d044de596904d06b5937d61; no voters:
I20260128 15:03:33.846099 24677 leader_election.cc:290] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:03:33.846199 24682 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:03:33.846391 24682 raft_consensus.cc:697] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [term 1 LEADER]: Becoming Leader. State: Replica: 35872c210d044de596904d06b5937d61, State: Running, Role: LEADER
I20260128 15:03:33.846696 24682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35872c210d044de596904d06b5937d61" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 40993 } }
I20260128 15:03:33.847123 24677 sys_catalog.cc:565] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:03:33.847757 24682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35872c210d044de596904d06b5937d61. Latest consensus state: current_term: 1 leader_uuid: "35872c210d044de596904d06b5937d61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35872c210d044de596904d06b5937d61" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 40993 } } }
I20260128 15:03:33.848376 24682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:33.848274 24684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35872c210d044de596904d06b5937d61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35872c210d044de596904d06b5937d61" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 40993 } } }
I20260128 15:03:33.848726 24684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61 [sys.catalog]: This master's current role is: LEADER
I20260128 15:03:33.848836 24691 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:03:33.852011 24691 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:03:33.856863 24691 catalog_manager.cc:1357] Generated new cluster ID: 26478e12d0aa48659cdf81485c571f79
I20260128 15:03:33.856945 24691 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:03:33.875428 24691 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:03:33.876384 24691 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:03:33.888383 24691 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61: Generated new TSK 0
I20260128 15:03:33.889230 24691 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:03:33.939659 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:40993
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:03:34.044709 24705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:34.044958 24705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:34.045025 24705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:34.048352 24705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:34.048434 24705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:34.048552 24705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:03:34.052814 24705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:40993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:34.053880 24705 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:34.054750 24705 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:34.061414 24710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:34.061404 24711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:34.061404 24713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:34.061751 24705 server_base.cc:1060] running on GCE node
I20260128 15:03:34.062129 24705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:34.062672 24705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:34.063818 24705 hybrid_clock.cc:648] HybridClock initialized: now 1769612614063804 us; error 30 us; skew 500 ppm
Jan 28 15:03:34 dist-test-slave-m28t krb5kdc[24593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612614, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:34.066347 24705 init.cc:377] Logged in from keytab as kudu/127.13.88.129@KRBTEST.COM (short username kudu)
I20260128 15:03:34.067487 24705 webserver.cc:492] Webserver started at http://127.13.88.129:35031/ using document root <none> and password file <none>
I20260128 15:03:34.068058 24705 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:34.068112 24705 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:34.068293 24705 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:34.070072 24705 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "4319d232c6ee4423874c9aa2572c46af"
format_stamp: "Formatted at 2026-01-28 15:03:34 on dist-test-slave-m28t"
server_key: "3ca740175b253d1271f18e97b4516784"
server_key_iv: "9bc56af58db3a63bdb9729eb526a8959"
server_key_version: "encryptionkey@0"
I20260128 15:03:34.070561 24705 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "4319d232c6ee4423874c9aa2572c46af"
format_stamp: "Formatted at 2026-01-28 15:03:34 on dist-test-slave-m28t"
server_key: "3ca740175b253d1271f18e97b4516784"
server_key_iv: "9bc56af58db3a63bdb9729eb526a8959"
server_key_version: "encryptionkey@0"
I20260128 15:03:34.073951 24705 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260128 15:03:34.076218 24720 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:34.077276 24705 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260128 15:03:34.077384 24705 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "4319d232c6ee4423874c9aa2572c46af"
format_stamp: "Formatted at 2026-01-28 15:03:34 on dist-test-slave-m28t"
server_key: "3ca740175b253d1271f18e97b4516784"
server_key_iv: "9bc56af58db3a63bdb9729eb526a8959"
server_key_version: "encryptionkey@0"
I20260128 15:03:34.077478 24705 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:34.097095 24705 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:34.099807 24705 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:34.099989 24705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:34.100574 24705 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:34.101485 24705 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:34.101533 24705 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:34.101585 24705 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:34.101613 24705 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:34.110546 24705 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:35225
I20260128 15:03:34.110574 24833 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:35225 every 8 connection(s)
I20260128 15:03:34.111562 24705 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:03:34.115022 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 24705
I20260128 15:03:34.115137 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:03:34.115356 13666 external_mini_cluster.cc:1468] Setting key 168d6a3d710f17385bdba4bd9e7b4dae
Jan 28 15:03:34 dist-test-slave-m28t krb5kdc[24593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612614, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.88.130@KRBTEST.COM; defaulting to no policy
I20260128 15:03:34.124473 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.113420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:42819 (local address 127.13.88.190:40993)
0128 15:03:34.113584 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:34.113587 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:34.114308 (+ 721us) server_negotiation.cc:408] Connection header received
0128 15:03:34.115211 (+ 903us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:34.115215 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:34.115276 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:34.115358 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:34.117298 (+ 1940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.117800 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.118413 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.118596 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.121031 (+ 2435us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:34.121056 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:34.121058 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:34.121088 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:34.122432 (+ 1344us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:34.123047 (+ 615us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:34.123050 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:34.123052 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:34.123090 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:34.123389 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:34.123391 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:34.123393 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:34.123558 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:34.123636 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:34.124147 (+ 511us) server_negotiation.cc:300] Negotiation successful
0128 15:03:34.124265 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260128 15:03:34.125356 24836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.113693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:40993 (local address 127.13.88.129:42819)
0128 15:03:34.114159 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:34.114198 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:34.114991 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:34.115553 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:34.115562 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:34.116044 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:34.117146 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:34.117162 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.117916 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.117919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:34.118300 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:34.118306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.118520 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.119110 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:34.119137 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:34.120837 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:34.122572 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:34.122579 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:34.122596 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:34.122955 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:34.123160 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:34.123163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:34.123164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:34.123300 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:34.123655 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:34.123666 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:34.123931 (+ 265us) client_negotiation.cc:769] Sending connection context
0128 15:03:34.124121 (+ 190us) client_negotiation.cc:240] Negotiation successful
0128 15:03:34.124406 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.130@KRBTEST.COM" created.
I20260128 15:03:34.126610 24834 heartbeater.cc:344] Connected to a master server at 127.13.88.190:40993
I20260128 15:03:34.126900 24834 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:34.127571 24834 heartbeater.cc:507] Master 127.13.88.190:40993 requested a full tablet report, sending...
I20260128 15:03:34.129129 24641 ts_manager.cc:194] Registered new tserver with Master: 4319d232c6ee4423874c9aa2572c46af (127.13.88.129:35225)
I20260128 15:03:34.130295 24641 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.129@KRBTEST.COM'} at 127.13.88.129:42819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:03:34.161686 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:40993
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:03:34.268976 24841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:34.269244 24841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:34.269321 24841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:34.272847 24841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:34.272931 24841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:34.273034 24841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:03:34.277664 24841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:40993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:34.278797 24841 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:34.279685 24841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:34.286664 24849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:34.286645 24846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:34.286645 24847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:34.286958 24841 server_base.cc:1060] running on GCE node
I20260128 15:03:34.287389 24841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:34.288004 24841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:34.289191 24841 hybrid_clock.cc:648] HybridClock initialized: now 1769612614289167 us; error 37 us; skew 500 ppm
Jan 28 15:03:34 dist-test-slave-m28t krb5kdc[24593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612614, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:34.291916 24841 init.cc:377] Logged in from keytab as kudu/127.13.88.130@KRBTEST.COM (short username kudu)
I20260128 15:03:34.293143 24841 webserver.cc:492] Webserver started at http://127.13.88.130:45575/ using document root <none> and password file <none>
I20260128 15:03:34.293700 24841 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:34.293756 24841 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:34.293960 24841 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:34.295680 24841 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "9b5856dc756946a78b3d1fe5fcf41c5b"
format_stamp: "Formatted at 2026-01-28 15:03:34 on dist-test-slave-m28t"
server_key: "f8e496465dc8dc31897bc06d04cbed58"
server_key_iv: "b2b706395e725b009e726b71a3508204"
server_key_version: "encryptionkey@0"
I20260128 15:03:34.296160 24841 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "9b5856dc756946a78b3d1fe5fcf41c5b"
format_stamp: "Formatted at 2026-01-28 15:03:34 on dist-test-slave-m28t"
server_key: "f8e496465dc8dc31897bc06d04cbed58"
server_key_iv: "b2b706395e725b009e726b71a3508204"
server_key_version: "encryptionkey@0"
I20260128 15:03:34.299688 24841 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260128 15:03:34.301857 24856 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:34.302852 24841 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:03:34.302959 24841 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "9b5856dc756946a78b3d1fe5fcf41c5b"
format_stamp: "Formatted at 2026-01-28 15:03:34 on dist-test-slave-m28t"
server_key: "f8e496465dc8dc31897bc06d04cbed58"
server_key_iv: "b2b706395e725b009e726b71a3508204"
server_key_version: "encryptionkey@0"
I20260128 15:03:34.303047 24841 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:34.319572 24841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:34.322328 24841 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:34.322511 24841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:34.323096 24841 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:34.324025 24841 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:34.324077 24841 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:34.324133 24841 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:34.324158 24841 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:34.332938 24841 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:33833
I20260128 15:03:34.333053 24969 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:33833 every 8 connection(s)
I20260128 15:03:34.334046 24841 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
I20260128 15:03:34.337070 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 24841
I20260128 15:03:34.337177 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:03:34.337410 13666 external_mini_cluster.cc:1468] Setting key d2cebc6c77e2f61ba351ea472ee1c772
Jan 28 15:03:34 dist-test-slave-m28t krb5kdc[24593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612614, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.88.131@KRBTEST.COM; defaulting to no policy
I20260128 15:03:34.347222 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.335828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:41013 (local address 127.13.88.190:40993)
0128 15:03:34.335961 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:34.335965 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:34.336811 (+ 846us) server_negotiation.cc:408] Connection header received
0128 15:03:34.337752 (+ 941us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:34.337756 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:34.337818 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:34.337894 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:34.339647 (+ 1753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.340224 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.340943 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.341128 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.343298 (+ 2170us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:34.343325 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:34.343329 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:34.343366 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:34.344961 (+ 1595us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:34.345489 (+ 528us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:34.345493 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:34.345495 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:34.345560 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:34.346041 (+ 481us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:34.346044 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:34.346046 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:34.346255 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:34.346349 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:34.346939 (+ 590us) server_negotiation.cc:300] Negotiation successful
0128 15:03:34.347084 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260128 15:03:34.347913 24972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.336119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:40993 (local address 127.13.88.130:41013)
0128 15:03:34.336647 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:34.336689 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:34.337533 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:34.338076 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:34.338086 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:34.338503 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:34.339482 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:34.339498 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.340360 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.340363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:34.340826 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:34.340835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.341031 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.341582 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:34.341610 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:34.343123 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:34.345100 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:34.345106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:34.345117 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:34.345371 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:34.345686 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:34.345690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:34.345692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:34.345843 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:34.346375 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:34.346381 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:34.346663 (+ 282us) client_negotiation.cc:769] Sending connection context
0128 15:03:34.346861 (+ 198us) client_negotiation.cc:240] Negotiation successful
0128 15:03:34.347122 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.131@KRBTEST.COM" created.
I20260128 15:03:34.349251 24970 heartbeater.cc:344] Connected to a master server at 127.13.88.190:40993
I20260128 15:03:34.349530 24970 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:34.350111 24970 heartbeater.cc:507] Master 127.13.88.190:40993 requested a full tablet report, sending...
I20260128 15:03:34.351341 24641 ts_manager.cc:194] Registered new tserver with Master: 9b5856dc756946a78b3d1fe5fcf41c5b (127.13.88.130:33833)
I20260128 15:03:34.351945 24641 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.130@KRBTEST.COM'} at 127.13.88.130:41013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:03:34.383145 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.131:0
--local_ip_for_outbound_sockets=127.13.88.131
--webserver_interface=127.13.88.131
--webserver_port=0
--tserver_master_addrs=127.13.88.190:40993
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:03:34.486678 24977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:03:34.486908 24977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:03:34.486969 24977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:03:34.490279 24977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:03:34.490342 24977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:03:34.490430 24977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.131
I20260128 15:03:34.494590 24977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.88.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:40993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.131
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:03:34.495617 24977 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:03:34.496472 24977 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:03:34.502974 24983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:34.503019 24985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:03:34.502974 24982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:03:34.503208 24977 server_base.cc:1060] running on GCE node
I20260128 15:03:34.503580 24977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:03:34.504168 24977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:03:34.505337 24977 hybrid_clock.cc:648] HybridClock initialized: now 1769612614505303 us; error 55 us; skew 500 ppm
Jan 28 15:03:34 dist-test-slave-m28t krb5kdc[24593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612614, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:03:34.508098 24977 init.cc:377] Logged in from keytab as kudu/127.13.88.131@KRBTEST.COM (short username kudu)
I20260128 15:03:34.509186 24977 webserver.cc:492] Webserver started at http://127.13.88.131:44899/ using document root <none> and password file <none>
I20260128 15:03:34.509714 24977 fs_manager.cc:362] Metadata directory not provided
I20260128 15:03:34.509763 24977 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:03:34.509936 24977 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260128 15:03:34.511685 24977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data/instance:
uuid: "f5cdd192d79a4bd2a75032272e3db6ad"
format_stamp: "Formatted at 2026-01-28 15:03:34 on dist-test-slave-m28t"
server_key: "595ea438b6409b8eee0b6c0ff23efce0"
server_key_iv: "9a3c0b68e2ab2e17827a76136e297199"
server_key_version: "encryptionkey@0"
I20260128 15:03:34.512351 24977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance:
uuid: "f5cdd192d79a4bd2a75032272e3db6ad"
format_stamp: "Formatted at 2026-01-28 15:03:34 on dist-test-slave-m28t"
server_key: "595ea438b6409b8eee0b6c0ff23efce0"
server_key_iv: "9a3c0b68e2ab2e17827a76136e297199"
server_key_version: "encryptionkey@0"
I20260128 15:03:34.515720 24977 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260128 15:03:34.518009 24992 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:34.519011 24977 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260128 15:03:34.519121 24977 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal
uuid: "f5cdd192d79a4bd2a75032272e3db6ad"
format_stamp: "Formatted at 2026-01-28 15:03:34 on dist-test-slave-m28t"
server_key: "595ea438b6409b8eee0b6c0ff23efce0"
server_key_iv: "9a3c0b68e2ab2e17827a76136e297199"
server_key_version: "encryptionkey@0"
I20260128 15:03:34.519213 24977 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:03:34.532764 24977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:03:34.535286 24977 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:03:34.535470 24977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:03:34.535986 24977 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:03:34.536953 24977 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:03:34.537001 24977 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:34.537053 24977 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:03:34.537082 24977 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:03:34.545548 24977 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.131:38545
I20260128 15:03:34.545574 25105 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.131:38545 every 8 connection(s)
I20260128 15:03:34.546476 24977 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/data/info.pb
I20260128 15:03:34.548316 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 24977
I20260128 15:03:34.548466 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal/instance
I20260128 15:03:34.548761 13666 external_mini_cluster.cc:1468] Setting key 73748e129c6ab1a4c4214625d814d6ca
Jan 28 15:03:34 dist-test-slave-m28t krb5kdc[24593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612614, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.131@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:03:34.558234 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.548190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.131:47617 (local address 127.13.88.190:40993)
0128 15:03:34.548309 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:34.548313 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:34.549275 (+ 962us) server_negotiation.cc:408] Connection header received
0128 15:03:34.550330 (+ 1055us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:34.550334 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:34.550388 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:34.550467 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:34.552144 (+ 1677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.552684 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.553240 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.553366 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.555346 (+ 1980us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:34.555363 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:34.555366 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:34.555390 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:34.556699 (+ 1309us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:34.557134 (+ 435us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:34.557137 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:34.557138 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:34.557177 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:34.557371 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:34.557373 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:34.557374 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:34.557511 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:34.557576 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:34.558015 (+ 439us) server_negotiation.cc:300] Negotiation successful
0128 15:03:34.558112 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260128 15:03:34.558919 25108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.548617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:40993 (local address 127.13.88.131:47617)
0128 15:03:34.549122 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:34.549160 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:34.550150 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:34.550626 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:34.550636 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:34.551073 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:03:34.552008 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:34.552021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.552771 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.552782 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:34.553160 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:34.553166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.553323 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.553828 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:03:34.553851 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:03:34.555212 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:03:34.556822 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:34.556827 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:34.556837 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:34.557063 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:34.557236 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:03:34.557237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:03:34.557239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:03:34.557315 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:03:34.557578 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:03:34.557583 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:03:34.557818 (+ 235us) client_negotiation.cc:769] Sending connection context
0128 15:03:34.557984 (+ 166us) client_negotiation.cc:240] Negotiation successful
0128 15:03:34.558189 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":139,"threads_started":1}
I20260128 15:03:34.559953 25106 heartbeater.cc:344] Connected to a master server at 127.13.88.190:40993
I20260128 15:03:34.560184 25106 heartbeater.cc:461] Registering TS with master...
I20260128 15:03:34.560708 25106 heartbeater.cc:507] Master 127.13.88.190:40993 requested a full tablet report, sending...
I20260128 15:03:34.561645 24640 ts_manager.cc:194] Registered new tserver with Master: f5cdd192d79a4bd2a75032272e3db6ad (127.13.88.131:38545)
I20260128 15:03:34.562191 24640 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.131@KRBTEST.COM'} at 127.13.88.131:47617
I20260128 15:03:34.563325 13666 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260128 15:03:34.571257 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.565362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55100 (local address 127.13.88.190:40993)
0128 15:03:34.565489 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:34.565493 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:34.565584 (+ 91us) server_negotiation.cc:408] Connection header received
0128 15:03:34.565706 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:34.565709 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:34.565748 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:34.565813 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:03:34.566591 (+ 778us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.567068 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.567668 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.567845 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.568818 (+ 973us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:03:34.568835 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:03:34.568838 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:03:34.568861 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:03:34.569877 (+ 1016us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:34.570210 (+ 333us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:34.570213 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:34.570215 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:34.570255 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:03:34.570456 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:03:34.570460 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:03:34.570461 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:03:34.570681 (+ 220us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:03:34.570800 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:34.571004 (+ 204us) server_negotiation.cc:300] Negotiation successful
0128 15:03:34.571108 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260128 15:03:34.573815 24640 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260128 15:03:34.576112 24640 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260128 15:03:34.588547 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.584187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:33833 (local address 127.0.0.1:43722)
0128 15:03:34.584590 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:34.584633 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:34.584776 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:34.585357 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:34.585363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:34.585388 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:34.585678 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:34.585687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.586942 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.586945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:34.588036 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:34.588046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.588136 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.588196 (+ 60us) client_negotiation.cc:769] Sending connection context
0128 15:03:34.588280 (+ 84us) client_negotiation.cc:240] Negotiation successful
0128 15:03:34.588370 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":100,"threads_started":1}
I20260128 15:03:34.588904 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.584897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:38545 (local address 127.0.0.1:47066)
0128 15:03:34.585396 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:34.585409 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:34.585472 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:34.586292 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:34.586295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:34.586313 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:34.586798 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:34.586806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.587717 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.587720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:34.588544 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:34.588554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.588651 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.588668 (+ 17us) client_negotiation.cc:769] Sending connection context
0128 15:03:34.588712 (+ 44us) client_negotiation.cc:240] Negotiation successful
0128 15:03:34.588767 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":96,"threads_started":1}
I20260128 15:03:34.589294 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.585182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:35225 (local address 127.0.0.1:55746)
0128 15:03:34.585929 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:34.585940 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:34.586026 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:34.586550 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:34.586553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:34.586587 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:34.586805 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:34.586810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.588129 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.588132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:34.588970 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:34.588980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.589072 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.589090 (+ 18us) client_negotiation.cc:769] Sending connection context
0128 15:03:34.589134 (+ 44us) client_negotiation.cc:240] Negotiation successful
0128 15:03:34.589188 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"spinlock_wait_cycles":9088,"thread_start_us":97,"threads_started":1}
I20260128 15:03:34.589444 25118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.584281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43722 (local address 127.13.88.130:33833)
0128 15:03:34.584904 (+ 623us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:34.584910 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:34.584929 (+ 19us) server_negotiation.cc:408] Connection header received
0128 15:03:34.584996 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:34.585000 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:34.585184 (+ 184us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:34.585830 (+ 646us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:34.585969 (+ 139us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.586817 (+ 848us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.588516 (+ 1699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.589046 (+ 530us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.589169 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:34.589227 (+ 58us) server_negotiation.cc:300] Negotiation successful
0128 15:03:34.589311 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":171,"threads_started":1}
I20260128 15:03:34.590207 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.585348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55746 (local address 127.13.88.129:35225)
0128 15:03:34.585798 (+ 450us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:34.585804 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:34.586016 (+ 212us) server_negotiation.cc:408] Connection header received
0128 15:03:34.586136 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:34.586140 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:34.586278 (+ 138us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:34.586407 (+ 129us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:34.586944 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.588010 (+ 1066us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.589513 (+ 1503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.589916 (+ 403us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.589995 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:34.590043 (+ 48us) server_negotiation.cc:300] Negotiation successful
0128 15:03:34.590101 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":93,"threads_started":1}
I20260128 15:03:34.590512 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.585584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47066 (local address 127.13.88.131:38545)
0128 15:03:34.585908 (+ 324us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:34.585914 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:34.585934 (+ 20us) server_negotiation.cc:408] Connection header received
0128 15:03:34.586015 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:34.586020 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:34.586170 (+ 150us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:34.586295 (+ 125us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:34.586918 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.587609 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.589074 (+ 1465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.590148 (+ 1074us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.590246 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:34.590305 (+ 59us) server_negotiation.cc:300] Negotiation successful
0128 15:03:34.590384 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":136,"threads_started":1}
I20260128 15:03:34.592381 24904 tablet_service.cc:1505] Processing CreateTablet for tablet 148098dc3eae4f08a757295bd74dadd1 (DEFAULT_TABLE table=test-table [id=a1ba5a98ae7f431cb86c1a6d96d9cdc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:34.592551 24768 tablet_service.cc:1505] Processing CreateTablet for tablet 148098dc3eae4f08a757295bd74dadd1 (DEFAULT_TABLE table=test-table [id=a1ba5a98ae7f431cb86c1a6d96d9cdc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:34.592823 25040 tablet_service.cc:1505] Processing CreateTablet for tablet 148098dc3eae4f08a757295bd74dadd1 (DEFAULT_TABLE table=test-table [id=a1ba5a98ae7f431cb86c1a6d96d9cdc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260128 15:03:34.593374 24768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 148098dc3eae4f08a757295bd74dadd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:34.593381 24904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 148098dc3eae4f08a757295bd74dadd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:34.593544 25040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 148098dc3eae4f08a757295bd74dadd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:03:34.598803 25123 tablet_bootstrap.cc:492] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af: Bootstrap starting.
I20260128 15:03:34.600171 25124 tablet_bootstrap.cc:492] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad: Bootstrap starting.
I20260128 15:03:34.600721 25125 tablet_bootstrap.cc:492] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b: Bootstrap starting.
I20260128 15:03:34.601408 25123 tablet_bootstrap.cc:654] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:34.601902 25124 tablet_bootstrap.cc:654] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:34.602243 25123 log.cc:826] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af: Log is configured to *not* fsync() on all Append() calls
I20260128 15:03:34.602520 25124 log.cc:826] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad: Log is configured to *not* fsync() on all Append() calls
I20260128 15:03:34.602595 25125 tablet_bootstrap.cc:654] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b: Neither blocks nor log segments found. Creating new log.
I20260128 15:03:34.603272 25125 log.cc:826] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b: Log is configured to *not* fsync() on all Append() calls
I20260128 15:03:34.604003 25123 tablet_bootstrap.cc:492] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af: No bootstrap required, opened a new log
I20260128 15:03:34.604179 25123 ts_tablet_manager.cc:1403] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260128 15:03:34.604913 25124 tablet_bootstrap.cc:492] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad: No bootstrap required, opened a new log
I20260128 15:03:34.604913 25125 tablet_bootstrap.cc:492] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b: No bootstrap required, opened a new log
I20260128 15:03:34.605088 25125 ts_tablet_manager.cc:1403] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260128 15:03:34.605088 25124 ts_tablet_manager.cc:1403] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260128 15:03:34.607143 25123 raft_consensus.cc:359] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5856dc756946a78b3d1fe5fcf41c5b" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 33833 } } peers { permanent_uuid: "f5cdd192d79a4bd2a75032272e3db6ad" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 38545 } } peers { permanent_uuid: "4319d232c6ee4423874c9aa2572c46af" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35225 } }
I20260128 15:03:34.607334 25123 raft_consensus.cc:385] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:34.607388 25123 raft_consensus.cc:740] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4319d232c6ee4423874c9aa2572c46af, State: Initialized, Role: FOLLOWER
I20260128 15:03:34.607786 25123 consensus_queue.cc:260] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5856dc756946a78b3d1fe5fcf41c5b" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 33833 } } peers { permanent_uuid: "f5cdd192d79a4bd2a75032272e3db6ad" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 38545 } } peers { permanent_uuid: "4319d232c6ee4423874c9aa2572c46af" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35225 } }
I20260128 15:03:34.608445 24834 heartbeater.cc:499] Master 127.13.88.190:40993 was elected leader, sending a full tablet report...
I20260128 15:03:34.608626 25123 ts_tablet_manager.cc:1434] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260128 15:03:34.608716 25124 raft_consensus.cc:359] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5856dc756946a78b3d1fe5fcf41c5b" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 33833 } } peers { permanent_uuid: "f5cdd192d79a4bd2a75032272e3db6ad" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 38545 } } peers { permanent_uuid: "4319d232c6ee4423874c9aa2572c46af" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35225 } }
I20260128 15:03:34.608917 25124 raft_consensus.cc:385] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:34.608973 25124 raft_consensus.cc:740] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5cdd192d79a4bd2a75032272e3db6ad, State: Initialized, Role: FOLLOWER
I20260128 15:03:34.609200 25125 raft_consensus.cc:359] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5856dc756946a78b3d1fe5fcf41c5b" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 33833 } } peers { permanent_uuid: "f5cdd192d79a4bd2a75032272e3db6ad" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 38545 } } peers { permanent_uuid: "4319d232c6ee4423874c9aa2572c46af" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35225 } }
I20260128 15:03:34.609377 25125 raft_consensus.cc:385] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:03:34.609419 25125 raft_consensus.cc:740] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5856dc756946a78b3d1fe5fcf41c5b, State: Initialized, Role: FOLLOWER
I20260128 15:03:34.609382 25124 consensus_queue.cc:260] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5856dc756946a78b3d1fe5fcf41c5b" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 33833 } } peers { permanent_uuid: "f5cdd192d79a4bd2a75032272e3db6ad" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 38545 } } peers { permanent_uuid: "4319d232c6ee4423874c9aa2572c46af" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35225 } }
I20260128 15:03:34.609834 25125 consensus_queue.cc:260] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5856dc756946a78b3d1fe5fcf41c5b" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 33833 } } peers { permanent_uuid: "f5cdd192d79a4bd2a75032272e3db6ad" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 38545 } } peers { permanent_uuid: "4319d232c6ee4423874c9aa2572c46af" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35225 } }
I20260128 15:03:34.609985 25106 heartbeater.cc:499] Master 127.13.88.190:40993 was elected leader, sending a full tablet report...
I20260128 15:03:34.610165 25124 ts_tablet_manager.cc:1434] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260128 15:03:34.610577 25125 ts_tablet_manager.cc:1434] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260128 15:03:34.610724 24970 heartbeater.cc:499] Master 127.13.88.190:40993 was elected leader, sending a full tablet report...
W20260128 15:03:34.612972 24835 tablet.cc:2378] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260128 15:03:34.754155 25131 raft_consensus.cc:493] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260128 15:03:34.754348 25131 raft_consensus.cc:515] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5856dc756946a78b3d1fe5fcf41c5b" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 33833 } } peers { permanent_uuid: "f5cdd192d79a4bd2a75032272e3db6ad" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 38545 } } peers { permanent_uuid: "4319d232c6ee4423874c9aa2572c46af" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35225 } }
I20260128 15:03:34.755448 25131 leader_election.cc:290] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5cdd192d79a4bd2a75032272e3db6ad (127.13.88.131:38545), 4319d232c6ee4423874c9aa2572c46af (127.13.88.129:35225)
I20260128 15:03:34.758206 24972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.755673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.131:38545 (local address 127.13.88.130:50543)
0128 15:03:34.755819 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:34.755837 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:34.755959 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:34.756285 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:34.756287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:34.756308 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:34.756563 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:34.756569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.757273 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.757277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:34.757849 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:34.757856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.757955 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.757977 (+ 22us) client_negotiation.cc:769] Sending connection context
0128 15:03:34.758023 (+ 46us) client_negotiation.cc:240] Negotiation successful
0128 15:03:34.758077 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260128 15:03:34.758615 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.755954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:35225 (local address 127.13.88.130:35745)
0128 15:03:34.756265 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:34.756277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0128 15:03:34.756362 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:03:34.756620 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:03:34.756622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:03:34.756638 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0128 15:03:34.756873 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:34.756879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.757562 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.757565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:03:34.758140 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:03:34.758147 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.758248 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.758351 (+ 103us) client_negotiation.cc:769] Sending connection context
0128 15:03:34.758403 (+ 52us) client_negotiation.cc:240] Negotiation successful
0128 15:03:34.758469 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":115,"threads_started":1}
I20260128 15:03:34.758642 25122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.755790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:50543 (local address 127.13.88.131:38545)
0128 15:03:34.755945 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:34.755949 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:34.755967 (+ 18us) server_negotiation.cc:408] Connection header received
0128 15:03:34.756078 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:34.756082 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:34.756135 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:34.756206 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:34.756679 (+ 473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.757167 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.757961 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.758401 (+ 440us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.758439 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:34.758486 (+ 47us) server_negotiation.cc:300] Negotiation successful
0128 15:03:34.758540 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260128 15:03:34.758925 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.756105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.130:35745 (local address 127.13.88.129:35225)
0128 15:03:34.756227 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:34.756230 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:34.756337 (+ 107us) server_negotiation.cc:408] Connection header received
0128 15:03:34.756489 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:34.756492 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:34.756537 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:34.756600 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0128 15:03:34.756989 (+ 389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.757457 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.758249 (+ 792us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.758719 (+ 470us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.758757 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:34.758799 (+ 42us) server_negotiation.cc:300] Negotiation successful
0128 15:03:34.758841 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260128 15:03:34.759274 25060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "148098dc3eae4f08a757295bd74dadd1" candidate_uuid: "9b5856dc756946a78b3d1fe5fcf41c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cdd192d79a4bd2a75032272e3db6ad" is_pre_election: true
I20260128 15:03:34.759444 24788 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "148098dc3eae4f08a757295bd74dadd1" candidate_uuid: "9b5856dc756946a78b3d1fe5fcf41c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4319d232c6ee4423874c9aa2572c46af" is_pre_election: true
I20260128 15:03:34.759536 25060 raft_consensus.cc:2468] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b5856dc756946a78b3d1fe5fcf41c5b in term 0.
I20260128 15:03:34.759675 24788 raft_consensus.cc:2468] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b5856dc756946a78b3d1fe5fcf41c5b in term 0.
I20260128 15:03:34.759981 24857 leader_election.cc:304] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b5856dc756946a78b3d1fe5fcf41c5b, f5cdd192d79a4bd2a75032272e3db6ad; no voters:
I20260128 15:03:34.760195 25131 raft_consensus.cc:2804] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260128 15:03:34.760275 25131 raft_consensus.cc:493] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260128 15:03:34.760326 25131 raft_consensus.cc:3060] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:34.761262 25131 raft_consensus.cc:515] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5856dc756946a78b3d1fe5fcf41c5b" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 33833 } } peers { permanent_uuid: "f5cdd192d79a4bd2a75032272e3db6ad" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 38545 } } peers { permanent_uuid: "4319d232c6ee4423874c9aa2572c46af" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35225 } }
I20260128 15:03:34.761576 25131 leader_election.cc:290] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [CANDIDATE]: Term 1 election: Requested vote from peers f5cdd192d79a4bd2a75032272e3db6ad (127.13.88.131:38545), 4319d232c6ee4423874c9aa2572c46af (127.13.88.129:35225)
I20260128 15:03:34.761902 25060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "148098dc3eae4f08a757295bd74dadd1" candidate_uuid: "9b5856dc756946a78b3d1fe5fcf41c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cdd192d79a4bd2a75032272e3db6ad"
I20260128 15:03:34.761938 24788 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "148098dc3eae4f08a757295bd74dadd1" candidate_uuid: "9b5856dc756946a78b3d1fe5fcf41c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4319d232c6ee4423874c9aa2572c46af"
I20260128 15:03:34.762008 25060 raft_consensus.cc:3060] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:34.762042 24788 raft_consensus.cc:3060] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:03:34.762907 25060 raft_consensus.cc:2468] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5856dc756946a78b3d1fe5fcf41c5b in term 1.
I20260128 15:03:34.762933 24788 raft_consensus.cc:2468] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5856dc756946a78b3d1fe5fcf41c5b in term 1.
I20260128 15:03:34.763146 24857 leader_election.cc:304] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4319d232c6ee4423874c9aa2572c46af, 9b5856dc756946a78b3d1fe5fcf41c5b; no voters:
I20260128 15:03:34.763298 25131 raft_consensus.cc:2804] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:03:34.763511 25131 raft_consensus.cc:697] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [term 1 LEADER]: Becoming Leader. State: Replica: 9b5856dc756946a78b3d1fe5fcf41c5b, State: Running, Role: LEADER
I20260128 15:03:34.763839 25131 consensus_queue.cc:237] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5856dc756946a78b3d1fe5fcf41c5b" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 33833 } } peers { permanent_uuid: "f5cdd192d79a4bd2a75032272e3db6ad" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 38545 } } peers { permanent_uuid: "4319d232c6ee4423874c9aa2572c46af" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35225 } }
I20260128 15:03:34.766961 24640 catalog_manager.cc:5654] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b5856dc756946a78b3d1fe5fcf41c5b (127.13.88.130). New cstate: current_term: 1 leader_uuid: "9b5856dc756946a78b3d1fe5fcf41c5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5856dc756946a78b3d1fe5fcf41c5b" member_type: VOTER last_known_addr { host: "127.13.88.130" port: 33833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5cdd192d79a4bd2a75032272e3db6ad" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 38545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4319d232c6ee4423874c9aa2572c46af" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35225 } health_report { overall_health: UNKNOWN } } }
I20260128 15:03:34.776688 25118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.773854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.13.88.130:33833)
0128 15:03:34.773971 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0128 15:03:34.773976 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:03:34.773989 (+ 13us) server_negotiation.cc:408] Connection header received
0128 15:03:34.774070 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:03:34.774073 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:03:34.774125 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:03:34.774204 (+ 79us) server_negotiation.cc:227] Negotiated authn=TOKEN
0128 15:03:34.774543 (+ 339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.775011 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:03:34.775612 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:03:34.775778 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:03:34.775866 (+ 88us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0128 15:03:34.776274 (+ 408us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0128 15:03:34.776381 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0128 15:03:34.776508 (+ 127us) server_negotiation.cc:300] Negotiation successful
0128 15:03:34.776561 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260128 15:03:34.781880 24788 raft_consensus.cc:1275] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af [term 1 FOLLOWER]: Refusing update from remote peer 9b5856dc756946a78b3d1fe5fcf41c5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:03:34.781878 25060 raft_consensus.cc:1275] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad [term 1 FOLLOWER]: Refusing update from remote peer 9b5856dc756946a78b3d1fe5fcf41c5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260128 15:03:34.782522 25131 consensus_queue.cc:1048] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5cdd192d79a4bd2a75032272e3db6ad" member_type: VOTER last_known_addr { host: "127.13.88.131" port: 38545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:03:34.782737 25133 consensus_queue.cc:1048] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4319d232c6ee4423874c9aa2572c46af" member_type: VOTER last_known_addr { host: "127.13.88.129" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260128 15:03:34.799477 24640 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:03:34.799649 24640 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260128 15:03:34.802078 24640 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 35872c210d044de596904d06b5937d61: Sending DeleteTablet for 3 replicas of tablet 148098dc3eae4f08a757295bd74dadd1
I20260128 15:03:34.802912 25040 tablet_service.cc:1552] Processing DeleteTablet for tablet 148098dc3eae4f08a757295bd74dadd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:34 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:47066
I20260128 15:03:34.802958 24768 tablet_service.cc:1552] Processing DeleteTablet for tablet 148098dc3eae4f08a757295bd74dadd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:34 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:55746
I20260128 15:03:34.803015 24904 tablet_service.cc:1552] Processing DeleteTablet for tablet 148098dc3eae4f08a757295bd74dadd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-28 15:03:34 UTC) from {username='kudu', principal='kudu/127.13.88.190@KRBTEST.COM'} at 127.0.0.1:43722
I20260128 15:03:34.803442 25148 tablet_replica.cc:333] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af: stopping tablet replica
I20260128 15:03:34.803846 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 24705
I20260128 15:03:34.803736 25148 raft_consensus.cc:2243] T 148098dc3eae4f08a757295bd74dadd1 P 4319d232c6ee4423874c9aa2572c46af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:03:34.805765 25146 tablet_replica.cc:333] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b: stopping tablet replica
I20260128 15:03:34.806026 25146 raft_consensus.cc:2243] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [term 1 LEADER]: Raft consensus shutting down.
I20260128 15:03:34.806430 25146 raft_consensus.cc:2272] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:34.807866 25146 ts_tablet_manager.cc:1916] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:34.810526 25146 ts_tablet_manager.cc:1929] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260128 15:03:34.810595 24625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.88.129:35225 (error 108)
I20260128 15:03:34.810736 25146 log.cc:1199] T 148098dc3eae4f08a757295bd74dadd1 P 9b5856dc756946a78b3d1fe5fcf41c5b: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-1/wal/wals/148098dc3eae4f08a757295bd74dadd1
I20260128 15:03:34.810801 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 24841
I20260128 15:03:34.812211 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.811899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.129:35225 (local address 127.0.0.1:55754)
0128 15:03:34.812070 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:34.812126 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.129:35225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260128 15:03:34.812470 24625 catalog_manager.cc:4712] TS 4319d232c6ee4423874c9aa2572c46af (127.13.88.129:35225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 148098dc3eae4f08a757295bd74dadd1: Network error: Client connection negotiation failed: client connection to 127.13.88.129:35225: connect: Connection refused (error 111)
I20260128 15:03:34.812597 25147 tablet_replica.cc:333] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad: stopping tablet replica
I20260128 15:03:34.812826 25147 raft_consensus.cc:2243] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260128 15:03:34.813097 25147 raft_consensus.cc:2272] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260128 15:03:34.814229 25147 ts_tablet_manager.cc:1916] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260128 15:03:34.816107 25147 ts_tablet_manager.cc:1929] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260128 15:03:34.816195 25147 log.cc:1199] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad: Deleting WAL directory at /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1769612519049359-13666-0/minicluster-data/ts-2/wal/wals/148098dc3eae4f08a757295bd74dadd1
I20260128 15:03:34.816517 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 24977
I20260128 15:03:34.816548 25147 ts_tablet_manager.cc:1950] T 148098dc3eae4f08a757295bd74dadd1 P f5cdd192d79a4bd2a75032272e3db6ad: Deleting consensus metadata
I20260128 15:03:34.817382 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:03:34.816707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.130:33833 (local address 127.0.0.1:43738)
0128 15:03:34.817248 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0128 15:03:34.817297 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.130:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460}
I20260128 15:03:34.821537 13666 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu with pid 24609
2026-01-28T15:03:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3126 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jan 28 15:03:34 dist-test-slave-m28t krb5kdc[25152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jan 28 15:03:34 dist-test-slave-m28t krb5kdc[25152](info): set up 2 sockets
Jan 28 15:03:34 dist-test-slave-m28t krb5kdc[25152](info): commencing operation
krb5kdc: starting...
W20260128 15:03:36.645020 13666 mini_kdc.cc:121] Time spent starting KDC: real 1.803s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/test-user.keytab.
Jan 28 15:03:36 dist-test-slave-m28t krb5kdc[25152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-01-28T15:03:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-01-28T15:03:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.88.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.88.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.88.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/rangeradmin_127.13.88.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.88.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/rangeradmin_127.13.88.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.88.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.88.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.88.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/rangerlookup_127.13.88.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.88.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/rangerlookup_127.13.88.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.88.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/HTTP_127.13.88.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.88.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/HTTP_127.13.88.148@KRBTEST.COM.keytab.
I20260128 15:03:36.793263 13666 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskoXbzrQ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/postgres -l logfile start
2026-01-28 15:03:39.336 UTC [25183] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-01-28 15:03:39.336 UTC [25183] LOG: listening on IPv4 address "127.13.88.148", port 51371
2026-01-28 15:03:39.340 UTC [25183] LOG: listening on Unix socket "/tmp/.s.PGSQL.51371"
2026-01-28 15:03:39.345 UTC [25188] LOG: database system was shut down at 2026-01-28 15:03:37 UTC
2026-01-28 15:03:39.350 UTC [25183] LOG: database system is ready to accept connections
I20260128 15:03:41.108667 13666 mini_postgres.cc:96] Postgres bound to 51371
2026-01-28 15:03:41.114 UTC [25194] FATAL: database "slave" does not exist
127.13.88.148:51371 - accepting connections
I20260128 15:03:41.114866 13666 mini_ranger.cc:161] Starting Ranger...
I20260128 15:03:41.132326 13666 mini_ranger.cc:84] Created miniranger Postgres user
I20260128 15:03:41.194739 13666 mini_ranger.cc:87] Created ranger Postgres database
I20260128 15:03:41.194854 13666 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-admin
2026-01-28 15:03:41,465 [I] DB FLAVOR :POSTGRES
2026-01-28 15:03:41,465 [I] --------- Verifying Ranger DB connection ---------
2026-01-28 15:03:41,466 [I] Checking connection..
2026-01-28 15:03:41,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-01-28 15:03:41,782 [I] Checking connection passed.
2026-01-28 15:03:41,783 [I] --------- Verifying version history table ---------
2026-01-28 15:03:41,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-28 15:03:42,083 [I] Table x_db_version_h does not exist in database ranger
2026-01-28 15:03:42,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-28 15:03:42,377 [I] Table x_db_version_h does not exist in database ranger
2026-01-28 15:03:42,377 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-01-28 15:03:42,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-01-28 15:03:42.686 UTC [25261] WARNING: there is no transaction in progress
2026-01-28 15:03:42,691 [I] create_dbversion_catalog.sql file imported successfully
2026-01-28 15:03:42,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-01-28 15:03:42,981 [I] Table x_db_version_h already exists in database 'ranger'
2026-01-28 15:03:42,981 [I] --------- Importing Ranger Core DB Schema ---------
2026-01-28 15:03:42,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-01-28 15:03:43,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-01-28 15:03:43,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m28t.c.gcp-upstream.internal','N') ;"
2026-01-28 15:03:43,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-01-28 15:03:44,173 [I] Table x_portal_user does not exist in database ranger
2026-01-28 15:03:44,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-01-28 15:03:44,467 [I] Table x_policy_ref_group does not exist in database ranger
2026-01-28 15:03:44,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-01-28 15:03:44,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-01-28 15:03:45,067 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-01-28 15:03:45,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-01-28 15:03:45.875 UTC [25378] WARNING: there is no transaction in progress
2026-01-28 15:03:45.890 UTC [25378] WARNING: there is no transaction in progress
2026-01-28 15:03:45.905 UTC [25378] WARNING: there is no transaction in progress
2026-01-28 15:03:45.916 UTC [25378] WARNING: there is no transaction in progress
2026-01-28 15:03:45.927 UTC [25378] WARNING: there is no transaction in progress
2026-01-28 15:03:45.940 UTC [25378] WARNING: there is no transaction in progress
2026-01-28 15:03:45.997 UTC [25378] WARNING: there is no transaction in progress
2026-01-28 15:03:46.003 UTC [25378] WARNING: there is no transaction in progress
2026-01-28 15:03:46.015 UTC [25378] WARNING: there is no transaction in progress
2026-01-28 15:03:46.028 UTC [25378] WARNING: there is no transaction in progress
2026-01-28 15:03:46.038 UTC [25378] WARNING: there is no transaction in progress
2026-01-28 15:03:46.049 UTC [25378] WARNING: there is no transaction in progress
2026-01-28 15:03:46.057 UTC [25378] WARNING: there is no transaction in progress
2026-01-28 15:03:46.062 UTC [25378] WARNING: there is no transaction in progress
2026-01-28 15:03:46.066 UTC [25378] WARNING: there is no transaction in progress
2026-01-28 15:03:46.070 UTC [25378] WARNING: there is no transaction in progress
2026-01-28 15:03:46,466 [I] ranger_core_db_postgres.sql file imported successfully
2026-01-28 15:03:46,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-01-28 15:03:46,758 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-01-28 15:03:46,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-01-28 15:03:47,072 [I] Table x_portal_user already exists in database 'ranger'
2026-01-28 15:03:47,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-01-28 15:03:47,391 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-01-28 15:03:47,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-01-28 15:03:47,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-01-28 15:03:48,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m28t.c.gcp-upstream.internal';"
2026-01-28 15:03:48,283 [I] CORE_DB_SCHEMA import status has been updated
2026-01-28 15:03:48,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-01-28 15:03:48,573 [I] DB_PATCHES have already been applied
I20260128 15:03:48.579053 13666 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-admin:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskoXbzrQ/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260128 15:03:48.579124 13666 mini_ranger.cc:193] Using host: 127.13.88.148
I20260128 15:03:48.582070 13666 mini_ranger.cc:227] Ranger admin URL: http://127.13.88.148:48343
Jan 28, 2026 3:03:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jan 28, 2026 3:03:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jan 28, 2026 3:03:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-admin/ews/webapp
Jan 28, 2026 3:03:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jan 28, 2026 3:03:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-admin/ews/webapp] .....
Jan 28, 2026 3:03:49 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Jan 28, 2026 3:03:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-admin/ews/webapp].
Jan 28, 2026 3:03:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Jan 28, 2026 3:03:49 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-48343"]
Jan 28, 2026 3:03:49 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Jan 28, 2026 3:03:49 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Jan 28, 2026 3:03:49 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Jan 28, 2026 3:03:49 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Jan 28, 2026 3:03:49 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Jan 28, 2026 3:03:50 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Jan 28, 2026 3:03:50 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jan 28, 2026 3:03:50 PM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Jan 28, 2026 3:03:50 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
I20260128 15:03:50.599131 13666 mini_ranger.cc:160] Time spent starting Ranger: real 9.484s user 0.005s sys 0.005s
Jan 28, 2026 3:04:10 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jan 28, 2026 3:04:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.AssetREST
Jan 28, 2026 3:04:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jan 28, 2026 3:04:10 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jan 28, 2026 3:04:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jan 28, 2026 3:04:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jan 28, 2026 3:04:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jan 28, 2026 3:04:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jan 28, 2026 3:04:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jan 28, 2026 3:04:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jan 28, 2026 3:04:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jan 28, 2026 3:04:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jan 28, 2026 3:04:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jan 28, 2026 3:04:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jan 28, 2026 3:04:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jan 28, 2026 3:04:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jan 28, 2026 3:04:10 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jan 28, 2026 3:04:11 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Jan 28, 2026 3:04:11 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-48343"]
I20260128 15:04:11.847451 13666 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.88.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.88.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.88.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/rangerkms_127.13.88.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.88.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/rangerkms_127.13.88.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.88.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.88.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/HTTP_127.13.88.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.88.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/HTTP_127.13.88.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jan 28 15:04:11 dist-test-slave-m28t krb5kdc[25152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612651, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260128 15:04:11.923437 13666 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260128 15:04:11.932900 13666 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260128 15:04:12.034770 13666 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260128 15:04:12.034951 13666 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-kms
I20260128 15:04:12.035069 13666 mini_ranger_kms.cc:227] Using postgres at 127.13.88.148:51371
2026-01-28 15:04:12,204 [I] DB FLAVOR :POSTGRES
2026-01-28 15:04:12,205 [I] --------- Verifying Ranger DB connection ---------
2026-01-28 15:04:12,205 [I] Checking connection
2026-01-28 15:04:12,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-01-28 15:04:12,511 [I] connection success
2026-01-28 15:04:12,511 [I] --------- Verifying Ranger DB tables ---------
2026-01-28 15:04:12,511 [I] Verifying table ranger_masterkey in database rangerkms
2026-01-28 15:04:12,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-01-28 15:04:12,813 [I] Table ranger_masterkey does not exist in database rangerkms
2026-01-28 15:04:12,814 [I] --------- Importing Ranger Core DB Schema ---------
2026-01-28 15:04:12,814 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-01-28 15:04:12,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.88.148:51371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-01-28 15:04:13,123 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260128 15:04:13.620486 13666 mini_ranger_kms.cc:310] Created kms service
I20260128 15:04:13.810171 13666 mini_ranger_kms.cc:326] Created kudu user
I20260128 15:04:13.853523 13666 mini_ranger_kms.cc:343] Created rangerkms user
Jan 28 15:04:13 dist-test-slave-m28t krb5kdc[25152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612651, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.88.148@KRBTEST.COM
I20260128 15:04:14.275168 13666 mini_ranger_kms.cc:384] Added ranger policy
I20260128 15:04:14.275368 13666 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-kms:/tmp/dist-test-taskoXbzrQ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskoXbzrQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260128 15:04:14.275394 13666 mini_ranger_kms.cc:240] Using host: 127.13.88.148
I20260128 15:04:14.279075 13666 mini_ranger_kms.cc:276] Ranger KMS PID: 25570
I20260128 15:04:14.279191 13666 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.88.148:48031
Jan 28, 2026 3:04:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jan 28, 2026 3:04:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jan 28, 2026 3:04:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jan 28, 2026 3:04:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-kms/ews/webapp] .....
Jan 28, 2026 3:04:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-kms/ews/webapp].
Jan 28, 2026 3:04:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jan 28, 2026 3:04:15 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-48031"]
Jan 28, 2026 3:04:15 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Jan 28, 2026 3:04:15 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Jan 28, 2026 3:04:15 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
I20260128 15:04:16.214886 13666 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.291s user 0.010s sys 0.142s
I20260128 15:04:16.215039 13666 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260128 15:04:16.215138 13666 mini_ranger_kms.cc:401] 127.13.88.148:48031/kms/v1/keys
Jan 28, 2026 3:04:16 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jan 28 15:04:16 dist-test-slave-m28t krb5kdc[25152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1769612656, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.88.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jan 28 15:04:16 dist-test-slave-m28t krb5kdc[25152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612656, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.88.148@KRBTEST.COM for HTTP/127.13.88.148@KRBTEST.COM
Jan 28 15:04:17 dist-test-slave-m28t krb5kdc[25152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612656, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.88.148@KRBTEST.COM for HTTP/127.13.88.148@KRBTEST.COM
Jan 28 15:04:19 dist-test-slave-m28t krb5kdc[25152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:04:19.412781 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:46785
--webserver_interface=127.13.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.88.190:46785
--ranger_config_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.88.148:48031/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:04:19.539013 25623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:04:19.539268 25623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:04:19.539352 25623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:04:19.542671 25623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260128 15:04:19.542740 25623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:04:19.542771 25623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:04:19.542794 25623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260128 15:04:19.542829 25623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260128 15:04:19.547289 25623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.88.148:48031/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.88.190:46785
--ranger_config_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.88.190:46785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.88.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:04:19.548343 25623 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:04:19.549235 25623 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260128 15:04:19.554982 25623 server_base.cc:1060] running on GCE node
W20260128 15:04:19.555119 25631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:04:19.555238 25628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:04:19.555074 25629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:04:19.555722 25623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:04:19.556679 25623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:04:19.557857 25623 hybrid_clock.cc:648] HybridClock initialized: now 1769612659557832 us; error 42 us; skew 500 ppm
Jan 28 15:04:19 dist-test-slave-m28t krb5kdc[25152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612659, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:04:19.560770 25623 init.cc:377] Logged in from keytab as kudu/127.13.88.190@KRBTEST.COM (short username kudu)
I20260128 15:04:19.562145 25623 webserver.cc:492] Webserver started at http://127.13.88.190:41505/ using document root <none> and password file <none>
I20260128 15:04:19.562852 25623 fs_manager.cc:362] Metadata directory not provided
I20260128 15:04:19.562920 25623 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:04:19.563138 25623 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 28 15:04:19 dist-test-slave-m28t krb5kdc[25152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612659, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.190@KRBTEST.COM for HTTP/127.13.88.148@KRBTEST.COM
I20260128 15:04:19.618688 25623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/data/instance:
uuid: "fe67685f9a2347068781002db49a7165"
format_stamp: "Formatted at 2026-01-28 15:04:19 on dist-test-slave-m28t"
server_key: "bc86ccb3ab5aa4397d8943f20965568d"
server_key_iv: "b2b68feabbe5d98486ea93aa18bc86a0"
server_key_version: "kuduclusterkey@0"
I20260128 15:04:19.619338 25623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/wal/instance:
uuid: "fe67685f9a2347068781002db49a7165"
format_stamp: "Formatted at 2026-01-28 15:04:19 on dist-test-slave-m28t"
server_key: "bc86ccb3ab5aa4397d8943f20965568d"
server_key_iv: "b2b68feabbe5d98486ea93aa18bc86a0"
server_key_version: "kuduclusterkey@0"
I20260128 15:04:19.623844 25623 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260128 15:04:19.641098 25638 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:04:19.642624 25623 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 15:04:19.642768 25623 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/wal
uuid: "fe67685f9a2347068781002db49a7165"
format_stamp: "Formatted at 2026-01-28 15:04:19 on dist-test-slave-m28t"
server_key: "bc86ccb3ab5aa4397d8943f20965568d"
server_key_iv: "b2b68feabbe5d98486ea93aa18bc86a0"
server_key_version: "kuduclusterkey@0"
I20260128 15:04:19.642902 25623 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:04:19.662420 25623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:04:19.665683 25623 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:04:19.665868 25623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:04:19.674849 25690 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.190:46785 every 8 connection(s)
I20260128 15:04:19.674850 25623 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.190:46785
I20260128 15:04:19.676244 25623 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/data/info.pb
I20260128 15:04:19.679361 25691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260128 15:04:19.680260 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 25623
I20260128 15:04:19.680445 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/wal/instance
I20260128 15:04:19.680732 13666 external_mini_cluster.cc:1468] Setting key 96ace69981708e1357a369d8234f7ca7
I20260128 15:04:19.685372 25691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165: Bootstrap starting.
I20260128 15:04:19.687935 25691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165: Neither blocks nor log segments found. Creating new log.
I20260128 15:04:19.688970 25691 log.cc:826] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165: Log is configured to *not* fsync() on all Append() calls
Jan 28 15:04:19 dist-test-slave-m28t krb5kdc[25152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:04:19.691176 25691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165: No bootstrap required, opened a new log
I20260128 15:04:19.693954 25691 raft_consensus.cc:359] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe67685f9a2347068781002db49a7165" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 46785 } }
I20260128 15:04:19.694201 25691 raft_consensus.cc:385] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260128 15:04:19.694283 25691 raft_consensus.cc:740] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe67685f9a2347068781002db49a7165, State: Initialized, Role: FOLLOWER
I20260128 15:04:19.694772 25691 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe67685f9a2347068781002db49a7165" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 46785 } }
I20260128 15:04:19.695056 25691 raft_consensus.cc:399] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260128 15:04:19.695181 25691 raft_consensus.cc:493] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260128 15:04:19.695282 25691 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [term 0 FOLLOWER]: Advancing to term 1
I20260128 15:04:19.696159 25691 raft_consensus.cc:515] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe67685f9a2347068781002db49a7165" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 46785 } }
I20260128 15:04:19.696174 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:04:19.682087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42650 (local address 127.13.88.190:46785)
0128 15:04:19.682657 (+ 570us) server_negotiation.cc:207] Beginning negotiation
0128 15:04:19.682670 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0128 15:04:19.682715 (+ 45us) server_negotiation.cc:408] Connection header received
0128 15:04:19.683403 (+ 688us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:04:19.683429 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:04:19.683813 (+ 384us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:04:19.684181 (+ 368us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:04:19.685243 (+ 1062us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:04:19.686497 (+ 1254us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:04:19.687366 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:04:19.687639 (+ 273us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:04:19.690397 (+ 2758us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:04:19.690422 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:04:19.690438 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:04:19.690489 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:04:19.693160 (+ 2671us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:04:19.693697 (+ 537us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:04:19.693703 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:04:19.693709 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:04:19.693796 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:04:19.694032 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:04:19.694036 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:04:19.694038 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:04:19.694505 (+ 467us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:04:19.694726 (+ 221us) server_negotiation.cc:1036] Waiting for connection context
0128 15:04:19.694978 (+ 252us) server_negotiation.cc:300] Negotiation successful
0128 15:04:19.695380 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":142,"threads_started":1}
I20260128 15:04:19.696586 25691 leader_election.cc:304] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe67685f9a2347068781002db49a7165; no voters:
I20260128 15:04:19.696908 25691 leader_election.cc:290] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260128 15:04:19.697005 25696 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [term 1 FOLLOWER]: Leader election won for term 1
I20260128 15:04:19.697219 25696 raft_consensus.cc:697] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [term 1 LEADER]: Becoming Leader. State: Replica: fe67685f9a2347068781002db49a7165, State: Running, Role: LEADER
I20260128 15:04:19.697558 25696 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe67685f9a2347068781002db49a7165" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 46785 } }
I20260128 15:04:19.698361 25691 sys_catalog.cc:565] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [sys.catalog]: configured and running, proceeding with master startup.
I20260128 15:04:19.698784 25696 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe67685f9a2347068781002db49a7165. Latest consensus state: current_term: 1 leader_uuid: "fe67685f9a2347068781002db49a7165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe67685f9a2347068781002db49a7165" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 46785 } } }
I20260128 15:04:19.698956 25696 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [sys.catalog]: This master's current role is: LEADER
I20260128 15:04:19.699133 25698 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe67685f9a2347068781002db49a7165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe67685f9a2347068781002db49a7165" member_type: VOTER last_known_addr { host: "127.13.88.190" port: 46785 } } }
I20260128 15:04:19.699328 25698 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165 [sys.catalog]: This master's current role is: LEADER
I20260128 15:04:19.700474 25700 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260128 15:04:19.700794 25691 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260128 15:04:19.703492 25700 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260128 15:04:19.719852 25700 catalog_manager.cc:1357] Generated new cluster ID: 436e771e2a3d428cae6a0abcea4cde83
I20260128 15:04:19.719956 25700 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260128 15:04:19.743980 25700 catalog_manager.cc:1380] Generated new certificate authority record
I20260128 15:04:19.745463 25700 catalog_manager.cc:1514] Loading token signing keys...
I20260128 15:04:19.758137 25700 catalog_manager.cc:6027] T 00000000000000000000000000000000 P fe67685f9a2347068781002db49a7165: Generated new TSK 0
I20260128 15:04:19.759217 25700 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab.
I20260128 15:04:20.613485 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.88.148:48031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.129:0
--local_ip_for_outbound_sockets=127.13.88.129
--webserver_interface=127.13.88.129
--webserver_port=0
--tserver_master_addrs=127.13.88.190:46785
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260128 15:04:20.741257 25737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:04:20.741511 25737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:04:20.741591 25737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:04:20.745136 25737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:04:20.745211 25737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:04:20.745321 25737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.129
I20260128 15:04:20.750721 25737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.88.148:48031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.88.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:46785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.129
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:04:20.752126 25737 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:04:20.752997 25737 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:04:20.759806 25744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:04:20.759806 25745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:04:20.760044 25737 server_base.cc:1060] running on GCE node
W20260128 15:04:20.759805 25747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:04:20.762168 25737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:04:20.762831 25737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:04:20.764003 25737 hybrid_clock.cc:648] HybridClock initialized: now 1769612660763983 us; error 53 us; skew 500 ppm
Jan 28 15:04:20 dist-test-slave-m28t krb5kdc[25152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612660, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:04:20.767210 25737 init.cc:377] Logged in from keytab as kudu/127.13.88.129@KRBTEST.COM (short username kudu)
I20260128 15:04:20.768334 25737 webserver.cc:492] Webserver started at http://127.13.88.129:41725/ using document root <none> and password file <none>
I20260128 15:04:20.768932 25737 fs_manager.cc:362] Metadata directory not provided
I20260128 15:04:20.768989 25737 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:04:20.769177 25737 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 28 15:04:20 dist-test-slave-m28t krb5kdc[25152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612660, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for HTTP/127.13.88.148@KRBTEST.COM
I20260128 15:04:20.785204 25737 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/data/instance:
uuid: "f61deaad1e0d44f08e3e5c40d1a079d7"
format_stamp: "Formatted at 2026-01-28 15:04:20 on dist-test-slave-m28t"
server_key: "e0743b5ad49d942af5de06c97dcecf43"
server_key_iv: "1d1506b09e3266a4e6cf95dd010e3c46"
server_key_version: "kuduclusterkey@0"
I20260128 15:04:20.785960 25737 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance:
uuid: "f61deaad1e0d44f08e3e5c40d1a079d7"
format_stamp: "Formatted at 2026-01-28 15:04:20 on dist-test-slave-m28t"
server_key: "e0743b5ad49d942af5de06c97dcecf43"
server_key_iv: "1d1506b09e3266a4e6cf95dd010e3c46"
server_key_version: "kuduclusterkey@0"
I20260128 15:04:20.794631 25737 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260128 15:04:20.814325 25754 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:04:20.815804 25737 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260128 15:04:20.816004 25737 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/wal
uuid: "f61deaad1e0d44f08e3e5c40d1a079d7"
format_stamp: "Formatted at 2026-01-28 15:04:20 on dist-test-slave-m28t"
server_key: "e0743b5ad49d942af5de06c97dcecf43"
server_key_iv: "1d1506b09e3266a4e6cf95dd010e3c46"
server_key_version: "kuduclusterkey@0"
I20260128 15:04:20.816207 25737 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:04:20.849511 25737 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:04:20.853240 25737 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:04:20.853506 25737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:04:20.854226 25737 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:04:20.855427 25737 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:04:20.855495 25737 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:04:20.855551 25737 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:04:20.855608 25737 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260128 15:04:20.869668 25737 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.129:37451
I20260128 15:04:20.870914 25737 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/data/info.pb
I20260128 15:04:20.873651 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 25737
I20260128 15:04:20.873782 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-0/wal/instance
I20260128 15:04:20.874042 13666 external_mini_cluster.cc:1468] Setting key ca5e1170feb7be00dff42ce357e4e569
I20260128 15:04:20.872205 25868 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.129:37451 every 8 connection(s)
WARNING: no policy specified for kudu/127.13.88.130@KRBTEST.COM; defaulting to no policy
Jan 28 15:04:20 dist-test-slave-m28t krb5kdc[25152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612660, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.129@KRBTEST.COM for kudu/127.13.88.190@KRBTEST.COM
I20260128 15:04:20.897279 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:04:20.874196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:46785 (local address 127.13.88.129:38953)
0128 15:04:20.875602 (+ 1406us) negotiation.cc:107] Waiting for socket to connect
0128 15:04:20.875649 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0128 15:04:20.876587 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0128 15:04:20.881578 (+ 4991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0128 15:04:20.881592 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0128 15:04:20.881967 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0128 15:04:20.882811 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:04:20.882832 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:04:20.887537 (+ 4705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0128 15:04:20.887543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0128 15:04:20.888222 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0128 15:04:20.888236 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0128 15:04:20.888522 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:04:20.889209 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0128 15:04:20.889252 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0128 15:04:20.890876 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0128 15:04:20.893686 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:04:20.893693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:04:20.893709 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:04:20.894140 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:04:20.894480 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0128 15:04:20.894483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0128 15:04:20.894486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0128 15:04:20.894636 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0128 15:04:20.895230 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0128 15:04:20.895239 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0128 15:04:20.895581 (+ 342us) client_negotiation.cc:769] Sending connection context
0128 15:04:20.895821 (+ 240us) client_negotiation.cc:240] Negotiation successful
0128 15:04:20.896195 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"thread_start_us":145,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.88.130@KRBTEST.COM" created.
I20260128 15:04:20.898859 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:04:20.875399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.88.129:38953 (local address 127.13.88.190:46785)
0128 15:04:20.881075 (+ 5676us) server_negotiation.cc:207] Beginning negotiation
0128 15:04:20.881080 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0128 15:04:20.881100 (+ 20us) server_negotiation.cc:408] Connection header received
0128 15:04:20.881148 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0128 15:04:20.881152 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0128 15:04:20.881213 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0128 15:04:20.881332 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0128 15:04:20.883024 (+ 1692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:04:20.883917 (+ 893us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0128 15:04:20.888924 (+ 5007us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0128 15:04:20.891222 (+ 2298us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0128 15:04:20.891367 (+ 145us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0128 15:04:20.891393 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0128 15:04:20.891396 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0128 15:04:20.891417 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0128 15:04:20.893525 (+ 2108us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:04:20.894302 (+ 777us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:04:20.894305 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:04:20.894308 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:04:20.894373 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0128 15:04:20.894752 (+ 379us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0128 15:04:20.894755 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0128 15:04:20.894757 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0128 15:04:20.894952 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0128 15:04:20.898469 (+ 3517us) server_negotiation.cc:1036] Waiting for connection context
0128 15:04:20.898556 (+ 87us) server_negotiation.cc:300] Negotiation successful
0128 15:04:20.898693 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5568,"thread_start_us":68,"threads_started":1}
I20260128 15:04:20.902215 25869 heartbeater.cc:344] Connected to a master server at 127.13.88.190:46785
I20260128 15:04:20.902565 25869 heartbeater.cc:461] Registering TS with master...
I20260128 15:04:20.903306 25869 heartbeater.cc:507] Master 127.13.88.190:46785 requested a full tablet report, sending...
I20260128 15:04:20.905128 25655 ts_manager.cc:194] Registered new tserver with Master: f61deaad1e0d44f08e3e5c40d1a079d7 (127.13.88.129:37451)
I20260128 15:04:20.906764 25655 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.88.129@KRBTEST.COM'} at 127.13.88.129:38953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.13.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.13.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab.
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.13.88.190 from keytab /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
I20260128 15:04:20.963981 13666 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
/tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.88.148:48031/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.88.130:0
--local_ip_for_outbound_sockets=127.13.88.130
--webserver_interface=127.13.88.130
--webserver_port=0
--tserver_master_addrs=127.13.88.190:46785
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.13.88.148:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
I20260128 15:04:20.981112 25729 server.cc:273] Received an EOF from the subprocess
F20260128 15:04:20.982096 25716 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7f3fc7b2adcd google::LogMessage::Fail() at ??:0
@ 0x7f3fc7b2eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f3fc7b2a7cc google::LogMessage::Flush() at ??:0
@ 0x7f3fc7b2bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3fc97e22d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7f3fc97df0d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7f3fc97e3005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x565460464a0a std::function<>::operator()() at ??:0
@ 0x7f3fc8f84b04 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3fc981e6db start_thread at ??:0
@ 0x7f3fc6ce171f clone at ??:0
W20260128 15:04:21.118470 25878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260128 15:04:21.118789 25878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260128 15:04:21.118871 25878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260128 15:04:21.124047 25878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260128 15:04:21.124147 25878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260128 15:04:21.124261 25878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.88.130
I20260128 15:04:21.130682 25878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.88.148:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.88.148:48031/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/krb5kdc/kudu.keytab
--principal=kudu/127.13.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.88.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.13.88.190:46785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.88.130
--log_dir=/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3722ed49c19ce54c7feae2fa944e32b1d4ed8d2d
build type DEBUG
built by None at 28 Jan 2026 14:43:15 UTC on 24a791456cd2
build id 10230
I20260128 15:04:21.132068 25878 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260128 15:04:21.133188 25878 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260128 15:04:21.142371 25898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:04:21.142565 25878 server_base.cc:1060] running on GCE node
W20260128 15:04:21.142761 25895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260128 15:04:21.142794 25896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260128 15:04:21.143401 25878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260128 15:04:21.144224 25878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260128 15:04:21.145423 25878 hybrid_clock.cc:648] HybridClock initialized: now 1769612661145410 us; error 65 us; skew 500 ppm
Jan 28 15:04:21 dist-test-slave-m28t krb5kdc[25152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612661, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260128 15:04:21.149047 25878 init.cc:377] Logged in from keytab as kudu/127.13.88.130@KRBTEST.COM (short username kudu)
I20260128 15:04:21.150478 25878 webserver.cc:492] Webserver started at http://127.13.88.130:36525/ using document root <none> and password file <none>
I20260128 15:04:21.151165 25878 fs_manager.cc:362] Metadata directory not provided
I20260128 15:04:21.151227 25878 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260128 15:04:21.151435 25878 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
Jan 28 15:04:21 dist-test-slave-m28t krb5kdc[25152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1769612661, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.88.130@KRBTEST.COM for HTTP/127.13.88.148@KRBTEST.COM
I20260128 15:04:21.166551 25878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/data/instance:
uuid: "5fd7b2798f60468cb73bd8a1ffa02a30"
format_stamp: "Formatted at 2026-01-28 15:04:21 on dist-test-slave-m28t"
server_key: "19e88c0f496f3d7fc1e6d572e4e8eaa6"
server_key_iv: "bb3cb63f0e4eb926269267fced8fcac4"
server_key_version: "kuduclusterkey@0"
I20260128 15:04:21.167272 25878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance:
uuid: "5fd7b2798f60468cb73bd8a1ffa02a30"
format_stamp: "Formatted at 2026-01-28 15:04:21 on dist-test-slave-m28t"
server_key: "19e88c0f496f3d7fc1e6d572e4e8eaa6"
server_key_iv: "bb3cb63f0e4eb926269267fced8fcac4"
server_key_version: "kuduclusterkey@0"
I20260128 15:04:21.173775 25878 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260128 15:04:21.189667 25905 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260128 15:04:21.191161 25878 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260128 15:04:21.191403 25878 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/wal
uuid: "5fd7b2798f60468cb73bd8a1ffa02a30"
format_stamp: "Formatted at 2026-01-28 15:04:21 on dist-test-slave-m28t"
server_key: "19e88c0f496f3d7fc1e6d572e4e8eaa6"
server_key_iv: "bb3cb63f0e4eb926269267fced8fcac4"
server_key_version: "kuduclusterkey@0"
I20260128 15:04:21.191636 25878 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260128 15:04:21.219182 25878 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260128 15:04:21.222074 25878 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260128 15:04:21.222258 25878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260128 15:04:21.222832 25878 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260128 15:04:21.223834 25878 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260128 15:04:21.223886 25878 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260128 15:04:21.223939 25878 ts_tablet_manager.cc:616] Registered 0 tablets
I20260128 15:04:21.223965 25878 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260128 15:04:21.234294 25878 rpc_server.cc:307] RPC server started. Bound to: 127.13.88.130:38227
I20260128 15:04:21.234549 26018 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.13.88.130:38227 every 8 connection(s)
I20260128 15:04:21.235378 25878 server_base.cc:1192] Dumped server information to /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/data/info.pb
I20260128 15:04:21.238785 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:04:21.237595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:46785 (local address 127.13.88.130:54085)
0128 15:04:21.238075 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0128 15:04:21.238176 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":113,"threads_started":1}
W20260128 15:04:21.239281 26019 heartbeater.cc:646] Failed to heartbeat to 127.13.88.190:46785 (0 consecutive failures): Network error: Failed to ping master at 127.13.88.190:46785: Client connection negotiation failed: client connection to 127.13.88.190:46785: connect: Connection refused (error 111)
I20260128 15:04:21.239979 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:04:21.239721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:46785 (local address 127.13.88.130:48169)
0128 15:04:21.239854 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0128 15:04:21.239908 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260128 15:04:21.240736 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:04:21.240539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:46785 (local address 127.13.88.130:40969)
0128 15:04:21.240640 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0128 15:04:21.240681 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260128 15:04:21.240914 26019 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260128 15:04:21.243240 13666 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskoXbzrQ/build/debug/bin/kudu as pid 25878
I20260128 15:04:21.243371 13666 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskoXbzrQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1769612519049359-13666-0/minicluster-data/ts-1/wal/instance
I20260128 15:04:21.243650 13666 external_mini_cluster.cc:1468] Setting key 33c2a62563451755ebccff58cec2c08c
I20260128 15:04:21.909830 25869 heartbeater.cc:499] Master 127.13.88.190:46785 was elected leader, sending a full tablet report...
I20260128 15:04:21.910837 26023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:04:21.910403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:46785 (local address 127.13.88.129:59453)
0128 15:04:21.910658 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0128 15:04:21.910729 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
W20260128 15:04:21.911111 25869 heartbeater.cc:646] Failed to heartbeat to 127.13.88.190:46785 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.88.190:46785: connect: Connection refused (error 111)
I20260128 15:04:22.241955 26024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:04:22.241544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:46785 (local address 127.13.88.130:47061)
0128 15:04:22.241790 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0128 15:04:22.241848 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260128 15:04:22.912293 26025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:04:22.911812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:46785 (local address 127.13.88.129:56153)
0128 15:04:22.912122 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0128 15:04:22.912186 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":93,"threads_started":1}
I20260128 15:04:23.243269 26026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:04:23.242837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:46785 (local address 127.13.88.130:57547)
0128 15:04:23.243091 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0128 15:04:23.243155 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260128 15:04:23.913653 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:04:23.913192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:46785 (local address 127.13.88.129:45011)
0128 15:04:23.913481 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0128 15:04:23.913541 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":103,"threads_started":1}
I20260128 15:04:24.244390 26028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:04:24.244000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:46785 (local address 127.13.88.130:50951)
0128 15:04:24.244240 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0128 15:04:24.244294 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20260128 15:04:24.914999 26029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:04:24.914520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:46785 (local address 127.13.88.129:43359)
0128 15:04:24.914834 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0128 15:04:24.914898 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":95,"threads_started":1}
I20260128 15:04:25.245620 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:04:25.245217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:46785 (local address 127.13.88.130:54509)
0128 15:04:25.245464 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0128 15:04:25.245522 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
I20260128 15:04:25.916317 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:04:25.915856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:46785 (local address 127.13.88.129:45769)
0128 15:04:25.916151 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0128 15:04:25.916216 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":80,"threads_started":1}
I20260128 15:04:26.246910 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0128 15:04:26.246468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.88.190:46785 (local address 127.13.88.130:42757)
0128 15:04:26.246752 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0128 15:04:26.246811 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.88.190:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":86,"threads_started":1}
F20260128 15:04:26.280531 13666 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1769612666 (unix time) try "date -d @1769612666" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800003562) received by PID 13666 (TID 0x7f3feefcfd80) from PID 13666; stack trace: ***
@ 0x7f3ffac76980 (unknown) at ??:0
@ 0x7f3ff82acfb7 gsignal at ??:0
@ 0x7f3ff82ae921 abort at ??:0
@ 0x7f3ff951adcd google::LogMessage::Fail() at ??:0
@ 0x7f3ff951eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f3ff951a7cc google::LogMessage::Flush() at ??:0
@ 0x7f3ff951bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3ffc2a7cab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f3ffc2a49c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x55e215eb392f kudu::SecurityITest::StartCluster() at ??:0
@ 0x55e215e9b792 kudu::SecurityITest_TestEncryptionWithKMSIntegration_Test::TestBody() at ??:0
@ 0x7f3ffa52b0ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f3ffa51fbdb testing::Test::Run() at ??:0
@ 0x7f3ffa51fd9d testing::TestInfo::Run() at ??:0
@ 0x7f3ffa520377 testing::TestSuite::Run() at ??:0
@ 0x7f3ffa52077c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f3ffa52b60d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f3ffa51fe63 testing::UnitTest::Run() at ??:0
@ 0x55e215eebc7e RUN_ALL_TESTS() at ??:0
@ 0x55e215eeae6b main at ??:0
@ 0x7f3ff828fbf7 __libc_start_main at ??:0
@ 0x55e215e8306a _start at ??:0
2026-01-28 15:04:26.460 UTC [25601] FATAL: terminating connection due to unexpected postmaster exit
2026-01-28 15:04:26.461 UTC [25494] FATAL: terminating connection due to unexpected postmaster exit
2026-01-28 15:04:26.461 UTC [25497] FATAL: terminating connection due to unexpected postmaster exit
2026-01-28 15:04:26.462 UTC [25493] FATAL: terminating connection due to unexpected postmaster exit
2026-01-28 15:04:26.463 UTC [25498] FATAL: terminating connection due to unexpected postmaster exit
2026-01-28 15:04:26.463 UTC [25495] FATAL: terminating connection due to unexpected postmaster exit